Note: This is test shard 2 of 8.
[==========] Running 8 tests from 6 test suites.
[----------] Global test environment set-up.
[----------] 1 test from RebalanceIgnoredTserversTest
[ RUN ] RebalanceIgnoredTserversTest.Basic
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240501 07:58:26.743285 398 test_util.cc:268] Using random seed: -637819957
I20240501 07:58:26.750378 398 ts_itest-base.cc:112] Starting cluster with:
I20240501 07:58:26.750447 398 ts_itest-base.cc:113] --------------
I20240501 07:58:26.750470 398 ts_itest-base.cc:114] 5 tablet servers
I20240501 07:58:26.750488 398 ts_itest-base.cc:115] 3 replicas per TS
I20240501 07:58:26.750504 398 ts_itest-base.cc:116] --------------
2024-05-01T07:58:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T07:58:26Z Disabled control of system clock
I20240501 07:58:26.760402 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tbgEt/build/debug/bin/kudu
/tmp/dist-test-task7tbgEt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714550306730830-398-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714550306730830-398-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714550306730830-398-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714550306730830-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.99.190:41853
--webserver_interface=127.0.99.190
--webserver_port=0
--builtin_ntp_servers=127.0.99.148:43961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.99.190:41853
--enable_range_replica_placement=false with env {}
W20240501 07:58:26.870460 410 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:58:26.870759 410 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 07:58:26.870832 410 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:58:26.874161 410 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 07:58:26.874243 410 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 07:58:26.874274 410 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 07:58:26.874306 410 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 07:58:26.878235 410 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:43961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714550306730830-398-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714550306730830-398-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.0.99.190:41853
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.99.190:41853
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714550306730830-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.0.99.190
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.410
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714550306730830-398-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 07:43:16 UTC on 1a15035a6830
build id 69
I20240501 07:58:26.879288 410 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 07:58:26.880492 410 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:58:26.888679 416 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:58:26.889793 410 server_base.cc:1034] running on GCE node
W20240501 07:58:26.890837 418 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:58:26.891542 415 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:58:26.892124 410 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 07:58:26.893191 410 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 07:58:26.894362 410 hybrid_clock.cc:648] HybridClock initialized: now 1714550306894314 us; error 65 us; skew 500 ppm
I20240501 07:58:26.900607 410 webserver.cc:460] Webserver started at http://127.0.99.190:36157/ using document root <none> and password file <none>
I20240501 07:58:26.901346 410 fs_manager.cc:359] Metadata directory not provided
I20240501 07:58:26.901419 410 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:58:26.901609 410 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 07:58:26.903439 410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714550306730830-398-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "17c1fd7f6d8b494888cd180b4e122bde"
format_stamp: "Formatted at 2024-05-01 07:58:26 on dist-test-slave-40g1"
I20240501 07:58:26.903952 410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714550306730830-398-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "17c1fd7f6d8b494888cd180b4e122bde"
format_stamp: "Formatted at 2024-05-01 07:58:26 on dist-test-slave-40g1"
I20240501 07:58:26.907634 410 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20240501 07:58:26.910022 424 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:26.911170 410 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240501 07:58:26.911314 410 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714550306730830-398-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714550306730830-398-0/raft_consensus-itest-cluster/master-0/wal
uuid: "17c1fd7f6d8b494888cd180b4e122bde"
format_stamp: "Formatted at 2024-05-01 07:58:26 on dist-test-slave-40g1"
I20240501 07:58:26.911419 410 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714550306730830-398-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714550306730830-398-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714550306730830-398-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:58:26.938510 410 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:58:26.939286 410 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 07:58:26.939471 410 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:58:26.946534 410 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.190:41853
I20240501 07:58:26.946548 476 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.190:41853 every 8 connection(s)
I20240501 07:58:26.947578 410 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714550306730830-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240501 07:58:26.950381 477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:26.955857 477 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17c1fd7f6d8b494888cd180b4e122bde: Bootstrap starting.
I20240501 07:58:26.956547 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tbgEt/build/debug/bin/kudu as pid 410
I20240501 07:58:26.956696 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714550306730830-398-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240501 07:58:26.958451 477 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 17c1fd7f6d8b494888cd180b4e122bde: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:26.959180 477 log.cc:824] T 00000000000000000000000000000000 P 17c1fd7f6d8b494888cd180b4e122bde: Log is configured to *not* fsync() on all Append() calls
I20240501 07:58:26.961254 477 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17c1fd7f6d8b494888cd180b4e122bde: No bootstrap required, opened a new log
I20240501 07:58:26.964385 477 raft_consensus.cc:348] T 00000000000000000000000000000000 P 17c1fd7f6d8b494888cd180b4e122bde [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17c1fd7f6d8b494888cd180b4e122bde" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 41853 } }
I20240501 07:58:26.964633 477 raft_consensus.cc:374] T 00000000000000000000000000000000 P 17c1fd7f6d8b494888cd180b4e122bde [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:26.964740 477 raft_consensus.cc:729] T 00000000000000000000000000000000 P 17c1fd7f6d8b494888cd180b4e122bde [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17c1fd7f6d8b494888cd180b4e122bde, State: Initialized, Role: FOLLOWER
I20240501 07:58:26.965251 477 consensus_queue.cc:260] T 00000000000000000000000000000000 P 17c1fd7f6d8b494888cd180b4e122bde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17c1fd7f6d8b494888cd180b4e122bde" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 41853 } }
I20240501 07:58:26.965382 477 raft_consensus.cc:388] T 00000000000000000000000000000000 P 17c1fd7f6d8b494888cd180b4e122bde [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 07:58:26.965440 477 raft_consensus.cc:482] T 00000000000000000000000000000000 P 17c1fd7f6d8b494888cd180b4e122bde [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 07:58:26.965541 477 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 17c1fd7f6d8b494888cd180b4e122bde [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:26.966595 477 raft_consensus.cc:504] T 00000000000000000000000000000000 P 17c1fd7f6d8b494888cd180b4e122bde [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17c1fd7f6d8b494888cd180b4e122bde" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 41853 } }
I20240501 07:58:26.966972 477 leader_election.cc:302] T 00000000000000000000000000000000 P 17c1fd7f6d8b494888cd180b4e122bde [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 17c1fd7f6d8b494888cd180b4e122bde; no voters:
I20240501 07:58:26.967305 477 leader_election.cc:288] T 00000000000000000000000000000000 P 17c1fd7f6d8b494888cd180b4e122bde [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 07:58:26.967646 482 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 17c1fd7f6d8b494888cd180b4e122bde [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:26.967902 482 raft_consensus.cc:686] T 00000000000000000000000000000000 P 17c1fd7f6d8b494888cd180b4e122bde [term 1 LEADER]: Becoming Leader. State: Replica: 17c1fd7f6d8b494888cd180b4e122bde, State: Running, Role: LEADER
I20240501 07:58:26.968261 482 consensus_queue.cc:237] T 00000000000000000000000000000000 P 17c1fd7f6d8b494888cd180b4e122bde [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17c1fd7f6d8b494888cd180b4e122bde" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 41853 } }
I20240501 07:58:26.968523 477 sys_catalog.cc:564] T 00000000000000000000000000000000 P 17c1fd7f6d8b494888cd180b4e122bde [sys.catalog]: configured and running, proceeding with master startup.
I20240501 07:58:26.969826 483 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17c1fd7f6d8b494888cd180b4e122bde [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "17c1fd7f6d8b494888cd180b4e122bde" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17c1fd7f6d8b494888cd180b4e122bde" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 41853 } } }
I20240501 07:58:26.969972 483 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17c1fd7f6d8b494888cd180b4e122bde [sys.catalog]: This master's current role is: LEADER
I20240501 07:58:26.970386 484 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17c1fd7f6d8b494888cd180b4e122bde [sys.catalog]: SysCatalogTable state changed. Reason: New leader 17c1fd7f6d8b494888cd180b4e122bde. Latest consensus state: current_term: 1 leader_uuid: "17c1fd7f6d8b494888cd180b4e122bde" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17c1fd7f6d8b494888cd180b4e122bde" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 41853 } } }
I20240501 07:58:26.970531 484 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17c1fd7f6d8b494888cd180b4e122bde [sys.catalog]: This master's current role is: LEADER
I20240501 07:58:26.971303 491 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 07:58:26.974054 491 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 07:58:26.979115 491 catalog_manager.cc:1347] Generated new cluster ID: 31c38e7fbec64005bf2fd7db7d45e0be
I20240501 07:58:26.979212 491 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 07:58:27.003680 491 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 07:58:27.004690 491 catalog_manager.cc:1504] Loading token signing keys...
I20240501 07:58:27.016081 491 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 17c1fd7f6d8b494888cd180b4e122bde: Generated new TSK 0
I20240501 07:58:27.017063 491 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 07:58:27.028455 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tbgEt/build/debug/bin/kudu
/tmp/dist-test-task7tbgEt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714550306730830-398-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714550306730830-398-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714550306730830-398-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714550306730830-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.129:0
--local_ip_for_outbound_sockets=127.0.99.129
--webserver_interface=127.0.99.129
--webserver_port=0
--tserver_master_addrs=127.0.99.190:41853
--builtin_ntp_servers=127.0.99.148:43961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20240501 07:58:27.142414 501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:58:27.142669 501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 07:58:27.142716 501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:58:27.146059 501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 07:58:27.146195 501 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.129
I20240501 07:58:27.150653 501 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:43961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714550306730830-398-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714550306730830-398-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714550306730830-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.0.99.129
--webserver_port=0
--tserver_master_addrs=127.0.99.190:41853
--never_fsync=true
--heap_profile_path=/tmp/kudu.501
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.129
--log_dir=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714550306730830-398-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 07:43:16 UTC on 1a15035a6830
build id 69
I20240501 07:58:27.151693 501 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 07:58:27.152873 501 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:58:27.161293 509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:58:27.162300 506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:58:27.163287 507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:58:27.164228 501 server_base.cc:1034] running on GCE node
I20240501 07:58:27.164693 501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 07:58:27.165388 501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 07:58:27.166553 501 hybrid_clock.cc:648] HybridClock initialized: now 1714550307166530 us; error 41 us; skew 500 ppm
I20240501 07:58:27.168664 501 webserver.cc:460] Webserver started at http://127.0.99.129:40969/ using document root <none> and password file <none>
I20240501 07:58:27.169339 501 fs_manager.cc:359] Metadata directory not provided
I20240501 07:58:27.169406 501 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:58:27.169603 501 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 07:58:27.171440 501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714550306730830-398-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "0a922c13fd4443bcb6469c550ced8f76"
format_stamp: "Formatted at 2024-05-01 07:58:27 on dist-test-slave-40g1"
I20240501 07:58:27.171941 501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714550306730830-398-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "0a922c13fd4443bcb6469c550ced8f76"
format_stamp: "Formatted at 2024-05-01 07:58:27 on dist-test-slave-40g1"
I20240501 07:58:27.175565 501 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20240501 07:58:27.177918 515 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:27.179062 501 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240501 07:58:27.179208 501 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714550306730830-398-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714550306730830-398-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "0a922c13fd4443bcb6469c550ced8f76"
format_stamp: "Formatted at 2024-05-01 07:58:27 on dist-test-slave-40g1"
I20240501 07:58:27.179306 501 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714550306730830-398-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714550306730830-398-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714550306730830-398-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:58:27.199154 501 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:58:27.199913 501 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 07:58:27.200089 501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:58:27.200701 501 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 07:58:27.201716 501 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 07:58:27.201771 501 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:27.201838 501 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 07:58:27.201864 501 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:27.211370 501 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.129:46413
I20240501 07:58:27.211391 628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.129:46413 every 8 connection(s)
I20240501 07:58:27.212421 501 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714550306730830-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240501 07:58:27.214593 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tbgEt/build/debug/bin/kudu as pid 501
I20240501 07:58:27.214704 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714550306730830-398-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240501 07:58:27.217094 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tbgEt/build/debug/bin/kudu
/tmp/dist-test-task7tbgEt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714550306730830-398-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714550306730830-398-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714550306730830-398-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714550306730830-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.130:0
--local_ip_for_outbound_sockets=127.0.99.130
--webserver_interface=127.0.99.130
--webserver_port=0
--tserver_master_addrs=127.0.99.190:41853
--builtin_ntp_servers=127.0.99.148:43961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20240501 07:58:27.223099 629 heartbeater.cc:347] Connected to a master server at 127.0.99.190:41853
I20240501 07:58:27.223352 629 heartbeater.cc:464] Registering TS with master...
I20240501 07:58:27.223910 629 heartbeater.cc:511] Master 127.0.99.190:41853 requested a full tablet report, sending...
I20240501 07:58:27.225555 441 ts_manager.cc:194] Registered new tserver with Master: 0a922c13fd4443bcb6469c550ced8f76 (127.0.99.129:46413)
I20240501 07:58:27.226737 441 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.129:59575
W20240501 07:58:27.332810 632 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:58:27.333070 632 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 07:58:27.333119 632 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:58:27.336302 632 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 07:58:27.336436 632 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.130
I20240501 07:58:27.340255 632 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:43961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714550306730830-398-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714550306730830-398-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714550306730830-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.0.99.130
--webserver_port=0
--tserver_master_addrs=127.0.99.190:41853
--never_fsync=true
--heap_profile_path=/tmp/kudu.632
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.130
--log_dir=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714550306730830-398-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 07:43:16 UTC on 1a15035a6830
build id 69
I20240501 07:58:27.341344 632 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 07:58:27.342535 632 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:58:27.351595 637 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:58:27.352589 640 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:58:27.353327 638 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:58:27.354202 632 server_base.cc:1034] running on GCE node
I20240501 07:58:27.354635 632 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 07:58:27.355229 632 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 07:58:27.356391 632 hybrid_clock.cc:648] HybridClock initialized: now 1714550307356376 us; error 34 us; skew 500 ppm
I20240501 07:58:27.358458 632 webserver.cc:460] Webserver started at http://127.0.99.130:34699/ using document root <none> and password file <none>
I20240501 07:58:27.359119 632 fs_manager.cc:359] Metadata directory not provided
I20240501 07:58:27.359180 632 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:58:27.359349 632 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 07:58:27.361114 632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714550306730830-398-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "492b4e4a628f4c8882e8eba3234db14e"
format_stamp: "Formatted at 2024-05-01 07:58:27 on dist-test-slave-40g1"
I20240501 07:58:27.361572 632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714550306730830-398-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "492b4e4a628f4c8882e8eba3234db14e"
format_stamp: "Formatted at 2024-05-01 07:58:27 on dist-test-slave-40g1"
I20240501 07:58:27.365036 632 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20240501 07:58:27.367398 646 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:27.368515 632 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 07:58:27.368649 632 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714550306730830-398-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714550306730830-398-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "492b4e4a628f4c8882e8eba3234db14e"
format_stamp: "Formatted at 2024-05-01 07:58:27 on dist-test-slave-40g1"
I20240501 07:58:27.368778 632 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714550306730830-398-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714550306730830-398-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714550306730830-398-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:58:27.382530 632 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:58:27.383278 632 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 07:58:27.383463 632 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:58:27.384089 632 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 07:58:27.385115 632 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 07:58:27.385170 632 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:27.385234 632 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 07:58:27.385267 632 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:27.395886 632 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.130:39683
I20240501 07:58:27.395926 759 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.130:39683 every 8 connection(s)
I20240501 07:58:27.396886 632 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714550306730830-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240501 07:58:27.403925 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tbgEt/build/debug/bin/kudu as pid 632
I20240501 07:58:27.404024 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714550306730830-398-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240501 07:58:27.405936 760 heartbeater.cc:347] Connected to a master server at 127.0.99.190:41853
I20240501 07:58:27.406189 760 heartbeater.cc:464] Registering TS with master...
I20240501 07:58:27.406307 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tbgEt/build/debug/bin/kudu
/tmp/dist-test-task7tbgEt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714550306730830-398-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714550306730830-398-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714550306730830-398-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714550306730830-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.131:0
--local_ip_for_outbound_sockets=127.0.99.131
--webserver_interface=127.0.99.131
--webserver_port=0
--tserver_master_addrs=127.0.99.190:41853
--builtin_ntp_servers=127.0.99.148:43961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20240501 07:58:27.406771 760 heartbeater.cc:511] Master 127.0.99.190:41853 requested a full tablet report, sending...
I20240501 07:58:27.407817 441 ts_manager.cc:194] Registered new tserver with Master: 492b4e4a628f4c8882e8eba3234db14e (127.0.99.130:39683)
I20240501 07:58:27.408460 441 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.130:55437
W20240501 07:58:27.523115 763 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:58:27.523365 763 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 07:58:27.523411 763 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:58:27.526716 763 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 07:58:27.526859 763 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.131
I20240501 07:58:27.531284 763 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:43961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714550306730830-398-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714550306730830-398-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714550306730830-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.0.99.131
--webserver_port=0
--tserver_master_addrs=127.0.99.190:41853
--never_fsync=true
--heap_profile_path=/tmp/kudu.763
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.131
--log_dir=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714550306730830-398-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 07:43:16 UTC on 1a15035a6830
build id 69
I20240501 07:58:27.532344 763 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 07:58:27.533607 763 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:58:27.542044 769 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:58:27.542922 768 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:58:27.544023 771 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:58:27.545040 763 server_base.cc:1034] running on GCE node
I20240501 07:58:27.545475 763 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 07:58:27.546095 763 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 07:58:27.547243 763 hybrid_clock.cc:648] HybridClock initialized: now 1714550307547219 us; error 31 us; skew 500 ppm
I20240501 07:58:27.549872 763 webserver.cc:460] Webserver started at http://127.0.99.131:35567/ using document root <none> and password file <none>
I20240501 07:58:27.550601 763 fs_manager.cc:359] Metadata directory not provided
I20240501 07:58:27.550673 763 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:58:27.550908 763 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 07:58:27.552990 763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714550306730830-398-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "34b0fe1b7b6448fcb991de294f4662d7"
format_stamp: "Formatted at 2024-05-01 07:58:27 on dist-test-slave-40g1"
I20240501 07:58:27.553635 763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714550306730830-398-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "34b0fe1b7b6448fcb991de294f4662d7"
format_stamp: "Formatted at 2024-05-01 07:58:27 on dist-test-slave-40g1"
I20240501 07:58:27.558188 763 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20240501 07:58:27.560523 777 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:27.561650 763 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 07:58:27.561751 763 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714550306730830-398-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714550306730830-398-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "34b0fe1b7b6448fcb991de294f4662d7"
format_stamp: "Formatted at 2024-05-01 07:58:27 on dist-test-slave-40g1"
I20240501 07:58:27.561836 763 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714550306730830-398-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714550306730830-398-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714550306730830-398-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:58:27.581861 763 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:58:27.583065 763 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 07:58:27.583228 763 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:58:27.583755 763 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 07:58:27.584654 763 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 07:58:27.584700 763 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:27.584741 763 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 07:58:27.584761 763 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:27.594007 763 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.131:39243
I20240501 07:58:27.594029 890 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.131:39243 every 8 connection(s)
I20240501 07:58:27.594976 763 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714550306730830-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240501 07:58:27.603665 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tbgEt/build/debug/bin/kudu as pid 763
I20240501 07:58:27.603911 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714550306730830-398-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240501 07:58:27.606496 891 heartbeater.cc:347] Connected to a master server at 127.0.99.190:41853
I20240501 07:58:27.606513 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tbgEt/build/debug/bin/kudu
/tmp/dist-test-task7tbgEt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714550306730830-398-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714550306730830-398-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714550306730830-398-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714550306730830-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.132:0
--local_ip_for_outbound_sockets=127.0.99.132
--webserver_interface=127.0.99.132
--webserver_port=0
--tserver_master_addrs=127.0.99.190:41853
--builtin_ntp_servers=127.0.99.148:43961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20240501 07:58:27.606786 891 heartbeater.cc:464] Registering TS with master...
I20240501 07:58:27.607573 891 heartbeater.cc:511] Master 127.0.99.190:41853 requested a full tablet report, sending...
I20240501 07:58:27.608703 441 ts_manager.cc:194] Registered new tserver with Master: 34b0fe1b7b6448fcb991de294f4662d7 (127.0.99.131:39243)
I20240501 07:58:27.609333 441 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.131:49259
W20240501 07:58:27.721743 894 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:58:27.721992 894 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 07:58:27.722035 894 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:58:27.725230 894 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 07:58:27.725349 894 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.132
I20240501 07:58:27.729238 894 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:43961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714550306730830-398-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714550306730830-398-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714550306730830-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.0.99.132
--webserver_port=0
--tserver_master_addrs=127.0.99.190:41853
--never_fsync=true
--heap_profile_path=/tmp/kudu.894
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.132
--log_dir=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714550306730830-398-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 07:43:16 UTC on 1a15035a6830
build id 69
I20240501 07:58:27.730211 894 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 07:58:27.731356 894 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:58:27.740497 902 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:58:27.740602 899 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:58:27.741329 900 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:58:27.742480 894 server_base.cc:1034] running on GCE node
I20240501 07:58:27.742884 894 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 07:58:27.743472 894 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 07:58:27.744649 894 hybrid_clock.cc:648] HybridClock initialized: now 1714550307744638 us; error 28 us; skew 500 ppm
I20240501 07:58:27.746677 894 webserver.cc:460] Webserver started at http://127.0.99.132:40089/ using document root <none> and password file <none>
I20240501 07:58:27.747277 894 fs_manager.cc:359] Metadata directory not provided
I20240501 07:58:27.747339 894 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:58:27.747524 894 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 07:58:27.749397 894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714550306730830-398-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "3e83693091294c9da92847195d436d01"
format_stamp: "Formatted at 2024-05-01 07:58:27 on dist-test-slave-40g1"
I20240501 07:58:27.749912 894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714550306730830-398-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "3e83693091294c9da92847195d436d01"
format_stamp: "Formatted at 2024-05-01 07:58:27 on dist-test-slave-40g1"
I20240501 07:58:27.753530 894 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240501 07:58:27.755834 908 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:27.757038 894 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240501 07:58:27.757151 894 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714550306730830-398-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714550306730830-398-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "3e83693091294c9da92847195d436d01"
format_stamp: "Formatted at 2024-05-01 07:58:27 on dist-test-slave-40g1"
I20240501 07:58:27.757252 894 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714550306730830-398-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714550306730830-398-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714550306730830-398-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:58:27.780623 894 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:58:27.781371 894 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 07:58:27.781536 894 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:58:27.782099 894 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 07:58:27.783077 894 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 07:58:27.783144 894 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:27.783214 894 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 07:58:27.783241 894 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:27.792892 894 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.132:39297
I20240501 07:58:27.792917 1021 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.132:39297 every 8 connection(s)
I20240501 07:58:27.794003 894 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714550306730830-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240501 07:58:27.803126 1022 heartbeater.cc:347] Connected to a master server at 127.0.99.190:41853
I20240501 07:58:27.803388 1022 heartbeater.cc:464] Registering TS with master...
I20240501 07:58:27.803393 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tbgEt/build/debug/bin/kudu as pid 894
I20240501 07:58:27.803494 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714550306730830-398-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240501 07:58:27.803993 1022 heartbeater.cc:511] Master 127.0.99.190:41853 requested a full tablet report, sending...
I20240501 07:58:27.805130 441 ts_manager.cc:194] Registered new tserver with Master: 3e83693091294c9da92847195d436d01 (127.0.99.132:39297)
I20240501 07:58:27.805662 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tbgEt/build/debug/bin/kudu
/tmp/dist-test-task7tbgEt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714550306730830-398-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714550306730830-398-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714550306730830-398-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714550306730830-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.133:0
--local_ip_for_outbound_sockets=127.0.99.133
--webserver_interface=127.0.99.133
--webserver_port=0
--tserver_master_addrs=127.0.99.190:41853
--builtin_ntp_servers=127.0.99.148:43961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20240501 07:58:27.805874 441 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.132:40055
W20240501 07:58:27.920125 1025 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:58:27.920383 1025 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 07:58:27.920428 1025 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:58:27.923666 1025 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 07:58:27.923789 1025 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.133
I20240501 07:58:27.927651 1025 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:43961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714550306730830-398-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714550306730830-398-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.133:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714550306730830-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.0.99.133
--webserver_port=0
--tserver_master_addrs=127.0.99.190:41853
--never_fsync=true
--heap_profile_path=/tmp/kudu.1025
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.133
--log_dir=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714550306730830-398-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 07:43:16 UTC on 1a15035a6830
build id 69
I20240501 07:58:27.928637 1025 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 07:58:27.929852 1025 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:58:27.939184 1033 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:58:27.939261 1030 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:58:27.940260 1031 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:58:27.941102 1025 server_base.cc:1034] running on GCE node
I20240501 07:58:27.941533 1025 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 07:58:27.942174 1025 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 07:58:27.943328 1025 hybrid_clock.cc:648] HybridClock initialized: now 1714550307943312 us; error 32 us; skew 500 ppm
I20240501 07:58:27.945279 1025 webserver.cc:460] Webserver started at http://127.0.99.133:36371/ using document root <none> and password file <none>
I20240501 07:58:27.945852 1025 fs_manager.cc:359] Metadata directory not provided
I20240501 07:58:27.945912 1025 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:58:27.946115 1025 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 07:58:27.947970 1025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714550306730830-398-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "d380f310313c4d66a4a7aa4fbf3ecd6c"
format_stamp: "Formatted at 2024-05-01 07:58:27 on dist-test-slave-40g1"
I20240501 07:58:27.948449 1025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714550306730830-398-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "d380f310313c4d66a4a7aa4fbf3ecd6c"
format_stamp: "Formatted at 2024-05-01 07:58:27 on dist-test-slave-40g1"
I20240501 07:58:27.952076 1025 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20240501 07:58:27.954578 1039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:27.955785 1025 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240501 07:58:27.955902 1025 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714550306730830-398-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714550306730830-398-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "d380f310313c4d66a4a7aa4fbf3ecd6c"
format_stamp: "Formatted at 2024-05-01 07:58:27 on dist-test-slave-40g1"
I20240501 07:58:27.956010 1025 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714550306730830-398-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714550306730830-398-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714550306730830-398-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:58:27.984586 1025 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:58:27.985335 1025 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 07:58:27.985486 1025 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:58:27.985994 1025 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 07:58:27.986886 1025 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 07:58:27.986932 1025 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:27.986972 1025 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 07:58:27.986992 1025 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:27.996148 1025 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.133:36945
I20240501 07:58:27.996150 1152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.133:36945 every 8 connection(s)
I20240501 07:58:27.997195 1025 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714550306730830-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 07:58:28.002357 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tbgEt/build/debug/bin/kudu as pid 1025
I20240501 07:58:28.002450 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714550306730830-398-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240501 07:58:28.006867 1153 heartbeater.cc:347] Connected to a master server at 127.0.99.190:41853
I20240501 07:58:28.007098 1153 heartbeater.cc:464] Registering TS with master...
I20240501 07:58:28.007611 1153 heartbeater.cc:511] Master 127.0.99.190:41853 requested a full tablet report, sending...
I20240501 07:58:28.008520 441 ts_manager.cc:194] Registered new tserver with Master: d380f310313c4d66a4a7aa4fbf3ecd6c (127.0.99.133:36945)
I20240501 07:58:28.009200 441 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.133:46293
I20240501 07:58:28.017338 398 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240501 07:58:28.031692 441 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59504:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240501 07:58:28.050365 825 tablet_service.cc:1467] Processing CreateTablet for tablet 3f5135a82e3742a590ced1958b11cd36 (DEFAULT_TABLE table=TestTable [id=762e1e5fbed4482ca0ddaa40479829ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:28.050316 1087 tablet_service.cc:1467] Processing CreateTablet for tablet 3f5135a82e3742a590ced1958b11cd36 (DEFAULT_TABLE table=TestTable [id=762e1e5fbed4482ca0ddaa40479829ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:28.051393 825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f5135a82e3742a590ced1958b11cd36. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:28.051393 1087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f5135a82e3742a590ced1958b11cd36. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:28.054876 956 tablet_service.cc:1467] Processing CreateTablet for tablet 3f5135a82e3742a590ced1958b11cd36 (DEFAULT_TABLE table=TestTable [id=762e1e5fbed4482ca0ddaa40479829ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:28.055727 956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f5135a82e3742a590ced1958b11cd36. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:28.057642 1171 tablet_bootstrap.cc:492] T 3f5135a82e3742a590ced1958b11cd36 P d380f310313c4d66a4a7aa4fbf3ecd6c: Bootstrap starting.
I20240501 07:58:28.058768 1172 tablet_bootstrap.cc:492] T 3f5135a82e3742a590ced1958b11cd36 P 34b0fe1b7b6448fcb991de294f4662d7: Bootstrap starting.
I20240501 07:58:28.060042 1171 tablet_bootstrap.cc:654] T 3f5135a82e3742a590ced1958b11cd36 P d380f310313c4d66a4a7aa4fbf3ecd6c: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:28.060834 1172 tablet_bootstrap.cc:654] T 3f5135a82e3742a590ced1958b11cd36 P 34b0fe1b7b6448fcb991de294f4662d7: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:28.061044 1171 log.cc:824] T 3f5135a82e3742a590ced1958b11cd36 P d380f310313c4d66a4a7aa4fbf3ecd6c: Log is configured to *not* fsync() on all Append() calls
I20240501 07:58:28.061630 1172 log.cc:824] T 3f5135a82e3742a590ced1958b11cd36 P 34b0fe1b7b6448fcb991de294f4662d7: Log is configured to *not* fsync() on all Append() calls
I20240501 07:58:28.061761 1173 tablet_bootstrap.cc:492] T 3f5135a82e3742a590ced1958b11cd36 P 3e83693091294c9da92847195d436d01: Bootstrap starting.
I20240501 07:58:28.062870 1171 tablet_bootstrap.cc:492] T 3f5135a82e3742a590ced1958b11cd36 P d380f310313c4d66a4a7aa4fbf3ecd6c: No bootstrap required, opened a new log
I20240501 07:58:28.063042 1171 ts_tablet_manager.cc:1367] T 3f5135a82e3742a590ced1958b11cd36 P d380f310313c4d66a4a7aa4fbf3ecd6c: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20240501 07:58:28.063712 1172 tablet_bootstrap.cc:492] T 3f5135a82e3742a590ced1958b11cd36 P 34b0fe1b7b6448fcb991de294f4662d7: No bootstrap required, opened a new log
I20240501 07:58:28.063761 1173 tablet_bootstrap.cc:654] T 3f5135a82e3742a590ced1958b11cd36 P 3e83693091294c9da92847195d436d01: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:28.063884 1172 ts_tablet_manager.cc:1367] T 3f5135a82e3742a590ced1958b11cd36 P 34b0fe1b7b6448fcb991de294f4662d7: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240501 07:58:28.064775 1173 log.cc:824] T 3f5135a82e3742a590ced1958b11cd36 P 3e83693091294c9da92847195d436d01: Log is configured to *not* fsync() on all Append() calls
I20240501 07:58:28.066669 1173 tablet_bootstrap.cc:492] T 3f5135a82e3742a590ced1958b11cd36 P 3e83693091294c9da92847195d436d01: No bootstrap required, opened a new log
I20240501 07:58:28.066830 1173 ts_tablet_manager.cc:1367] T 3f5135a82e3742a590ced1958b11cd36 P 3e83693091294c9da92847195d436d01: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240501 07:58:28.067009 1171 raft_consensus.cc:348] T 3f5135a82e3742a590ced1958b11cd36 P d380f310313c4d66a4a7aa4fbf3ecd6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34b0fe1b7b6448fcb991de294f4662d7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39243 } } peers { permanent_uuid: "d380f310313c4d66a4a7aa4fbf3ecd6c" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36945 } } peers { permanent_uuid: "3e83693091294c9da92847195d436d01" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39297 } }
I20240501 07:58:28.067199 1171 raft_consensus.cc:374] T 3f5135a82e3742a590ced1958b11cd36 P d380f310313c4d66a4a7aa4fbf3ecd6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:28.067255 1171 raft_consensus.cc:729] T 3f5135a82e3742a590ced1958b11cd36 P d380f310313c4d66a4a7aa4fbf3ecd6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d380f310313c4d66a4a7aa4fbf3ecd6c, State: Initialized, Role: FOLLOWER
I20240501 07:58:28.067548 1172 raft_consensus.cc:348] T 3f5135a82e3742a590ced1958b11cd36 P 34b0fe1b7b6448fcb991de294f4662d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34b0fe1b7b6448fcb991de294f4662d7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39243 } } peers { permanent_uuid: "d380f310313c4d66a4a7aa4fbf3ecd6c" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36945 } } peers { permanent_uuid: "3e83693091294c9da92847195d436d01" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39297 } }
I20240501 07:58:28.067648 1171 consensus_queue.cc:260] T 3f5135a82e3742a590ced1958b11cd36 P d380f310313c4d66a4a7aa4fbf3ecd6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34b0fe1b7b6448fcb991de294f4662d7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39243 } } peers { permanent_uuid: "d380f310313c4d66a4a7aa4fbf3ecd6c" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36945 } } peers { permanent_uuid: "3e83693091294c9da92847195d436d01" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39297 } }
I20240501 07:58:28.067740 1172 raft_consensus.cc:374] T 3f5135a82e3742a590ced1958b11cd36 P 34b0fe1b7b6448fcb991de294f4662d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:28.067826 1172 raft_consensus.cc:729] T 3f5135a82e3742a590ced1958b11cd36 P 34b0fe1b7b6448fcb991de294f4662d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34b0fe1b7b6448fcb991de294f4662d7, State: Initialized, Role: FOLLOWER
I20240501 07:58:28.068253 1172 consensus_queue.cc:260] T 3f5135a82e3742a590ced1958b11cd36 P 34b0fe1b7b6448fcb991de294f4662d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34b0fe1b7b6448fcb991de294f4662d7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39243 } } peers { permanent_uuid: "d380f310313c4d66a4a7aa4fbf3ecd6c" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36945 } } peers { permanent_uuid: "3e83693091294c9da92847195d436d01" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39297 } }
I20240501 07:58:28.068449 1171 ts_tablet_manager.cc:1398] T 3f5135a82e3742a590ced1958b11cd36 P d380f310313c4d66a4a7aa4fbf3ecd6c: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20240501 07:58:28.068821 1153 heartbeater.cc:503] Master 127.0.99.190:41853 was elected leader, sending a full tablet report...
I20240501 07:58:28.069080 1172 ts_tablet_manager.cc:1398] T 3f5135a82e3742a590ced1958b11cd36 P 34b0fe1b7b6448fcb991de294f4662d7: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20240501 07:58:28.069267 891 heartbeater.cc:503] Master 127.0.99.190:41853 was elected leader, sending a full tablet report...
I20240501 07:58:28.070457 1173 raft_consensus.cc:348] T 3f5135a82e3742a590ced1958b11cd36 P 3e83693091294c9da92847195d436d01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34b0fe1b7b6448fcb991de294f4662d7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39243 } } peers { permanent_uuid: "d380f310313c4d66a4a7aa4fbf3ecd6c" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36945 } } peers { permanent_uuid: "3e83693091294c9da92847195d436d01" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39297 } }
I20240501 07:58:28.070607 1173 raft_consensus.cc:374] T 3f5135a82e3742a590ced1958b11cd36 P 3e83693091294c9da92847195d436d01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:28.070637 1173 raft_consensus.cc:729] T 3f5135a82e3742a590ced1958b11cd36 P 3e83693091294c9da92847195d436d01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e83693091294c9da92847195d436d01, State: Initialized, Role: FOLLOWER
I20240501 07:58:28.070940 1173 consensus_queue.cc:260] T 3f5135a82e3742a590ced1958b11cd36 P 3e83693091294c9da92847195d436d01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34b0fe1b7b6448fcb991de294f4662d7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39243 } } peers { permanent_uuid: "d380f310313c4d66a4a7aa4fbf3ecd6c" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36945 } } peers { permanent_uuid: "3e83693091294c9da92847195d436d01" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39297 } }
I20240501 07:58:28.071483 1022 heartbeater.cc:503] Master 127.0.99.190:41853 was elected leader, sending a full tablet report...
I20240501 07:58:28.071858 1173 ts_tablet_manager.cc:1398] T 3f5135a82e3742a590ced1958b11cd36 P 3e83693091294c9da92847195d436d01: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
W20240501 07:58:28.096102 892 tablet.cc:2369] T 3f5135a82e3742a590ced1958b11cd36 P 34b0fe1b7b6448fcb991de294f4662d7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 07:58:28.114039 1177 raft_consensus.cc:482] T 3f5135a82e3742a590ced1958b11cd36 P d380f310313c4d66a4a7aa4fbf3ecd6c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:28.114261 1177 raft_consensus.cc:504] T 3f5135a82e3742a590ced1958b11cd36 P d380f310313c4d66a4a7aa4fbf3ecd6c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34b0fe1b7b6448fcb991de294f4662d7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39243 } } peers { permanent_uuid: "d380f310313c4d66a4a7aa4fbf3ecd6c" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36945 } } peers { permanent_uuid: "3e83693091294c9da92847195d436d01" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39297 } }
I20240501 07:58:28.115521 1177 leader_election.cc:288] T 3f5135a82e3742a590ced1958b11cd36 P d380f310313c4d66a4a7aa4fbf3ecd6c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 34b0fe1b7b6448fcb991de294f4662d7 (127.0.99.131:39243), 3e83693091294c9da92847195d436d01 (127.0.99.132:39297)
I20240501 07:58:28.119702 845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f5135a82e3742a590ced1958b11cd36" candidate_uuid: "d380f310313c4d66a4a7aa4fbf3ecd6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34b0fe1b7b6448fcb991de294f4662d7" is_pre_election: true
I20240501 07:58:28.119845 974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f5135a82e3742a590ced1958b11cd36" candidate_uuid: "d380f310313c4d66a4a7aa4fbf3ecd6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e83693091294c9da92847195d436d01" is_pre_election: true
I20240501 07:58:28.120071 845 raft_consensus.cc:2454] T 3f5135a82e3742a590ced1958b11cd36 P 34b0fe1b7b6448fcb991de294f4662d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d380f310313c4d66a4a7aa4fbf3ecd6c in term 0.
I20240501 07:58:28.120152 974 raft_consensus.cc:2454] T 3f5135a82e3742a590ced1958b11cd36 P 3e83693091294c9da92847195d436d01 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d380f310313c4d66a4a7aa4fbf3ecd6c in term 0.
I20240501 07:58:28.120563 1041 leader_election.cc:302] T 3f5135a82e3742a590ced1958b11cd36 P d380f310313c4d66a4a7aa4fbf3ecd6c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 34b0fe1b7b6448fcb991de294f4662d7, d380f310313c4d66a4a7aa4fbf3ecd6c; no voters:
I20240501 07:58:28.120844 1177 raft_consensus.cc:2781] T 3f5135a82e3742a590ced1958b11cd36 P d380f310313c4d66a4a7aa4fbf3ecd6c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:28.120918 1177 raft_consensus.cc:482] T 3f5135a82e3742a590ced1958b11cd36 P d380f310313c4d66a4a7aa4fbf3ecd6c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:28.120947 1177 raft_consensus.cc:3037] T 3f5135a82e3742a590ced1958b11cd36 P d380f310313c4d66a4a7aa4fbf3ecd6c [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:28.121862 1177 raft_consensus.cc:504] T 3f5135a82e3742a590ced1958b11cd36 P d380f310313c4d66a4a7aa4fbf3ecd6c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34b0fe1b7b6448fcb991de294f4662d7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39243 } } peers { permanent_uuid: "d380f310313c4d66a4a7aa4fbf3ecd6c" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36945 } } peers { permanent_uuid: "3e83693091294c9da92847195d436d01" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39297 } }
I20240501 07:58:28.122215 1177 leader_election.cc:288] T 3f5135a82e3742a590ced1958b11cd36 P d380f310313c4d66a4a7aa4fbf3ecd6c [CANDIDATE]: Term 1 election: Requested vote from peers 34b0fe1b7b6448fcb991de294f4662d7 (127.0.99.131:39243), 3e83693091294c9da92847195d436d01 (127.0.99.132:39297)
I20240501 07:58:28.122581 845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f5135a82e3742a590ced1958b11cd36" candidate_uuid: "d380f310313c4d66a4a7aa4fbf3ecd6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34b0fe1b7b6448fcb991de294f4662d7"
I20240501 07:58:28.122612 974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f5135a82e3742a590ced1958b11cd36" candidate_uuid: "d380f310313c4d66a4a7aa4fbf3ecd6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e83693091294c9da92847195d436d01"
I20240501 07:58:28.122718 974 raft_consensus.cc:3037] T 3f5135a82e3742a590ced1958b11cd36 P 3e83693091294c9da92847195d436d01 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:28.122718 845 raft_consensus.cc:3037] T 3f5135a82e3742a590ced1958b11cd36 P 34b0fe1b7b6448fcb991de294f4662d7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:28.123807 974 raft_consensus.cc:2454] T 3f5135a82e3742a590ced1958b11cd36 P 3e83693091294c9da92847195d436d01 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d380f310313c4d66a4a7aa4fbf3ecd6c in term 1.
I20240501 07:58:28.123807 845 raft_consensus.cc:2454] T 3f5135a82e3742a590ced1958b11cd36 P 34b0fe1b7b6448fcb991de294f4662d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d380f310313c4d66a4a7aa4fbf3ecd6c in term 1.
I20240501 07:58:28.124122 1041 leader_election.cc:302] T 3f5135a82e3742a590ced1958b11cd36 P d380f310313c4d66a4a7aa4fbf3ecd6c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 34b0fe1b7b6448fcb991de294f4662d7, d380f310313c4d66a4a7aa4fbf3ecd6c; no voters:
I20240501 07:58:28.124269 1177 raft_consensus.cc:2781] T 3f5135a82e3742a590ced1958b11cd36 P d380f310313c4d66a4a7aa4fbf3ecd6c [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:28.124481 1177 raft_consensus.cc:686] T 3f5135a82e3742a590ced1958b11cd36 P d380f310313c4d66a4a7aa4fbf3ecd6c [term 1 LEADER]: Becoming Leader. State: Replica: d380f310313c4d66a4a7aa4fbf3ecd6c, State: Running, Role: LEADER
I20240501 07:58:28.124821 1177 consensus_queue.cc:237] T 3f5135a82e3742a590ced1958b11cd36 P d380f310313c4d66a4a7aa4fbf3ecd6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34b0fe1b7b6448fcb991de294f4662d7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39243 } } peers { permanent_uuid: "d380f310313c4d66a4a7aa4fbf3ecd6c" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36945 } } peers { permanent_uuid: "3e83693091294c9da92847195d436d01" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39297 } }
I20240501 07:58:28.127872 440 catalog_manager.cc:5484] T 3f5135a82e3742a590ced1958b11cd36 P d380f310313c4d66a4a7aa4fbf3ecd6c reported cstate change: term changed from 0 to 1, leader changed from <none> to d380f310313c4d66a4a7aa4fbf3ecd6c (127.0.99.133). New cstate: current_term: 1 leader_uuid: "d380f310313c4d66a4a7aa4fbf3ecd6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34b0fe1b7b6448fcb991de294f4662d7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39243 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d380f310313c4d66a4a7aa4fbf3ecd6c" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 36945 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3e83693091294c9da92847195d436d01" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39297 } health_report { overall_health: UNKNOWN } } }
I20240501 07:58:28.144429 398 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240501 07:58:28.146123 398 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 0a922c13fd4443bcb6469c550ced8f76 to finish bootstrapping
I20240501 07:58:28.152594 398 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 492b4e4a628f4c8882e8eba3234db14e to finish bootstrapping
I20240501 07:58:28.158545 398 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 34b0fe1b7b6448fcb991de294f4662d7 to finish bootstrapping
I20240501 07:58:28.163697 398 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 3e83693091294c9da92847195d436d01 to finish bootstrapping
I20240501 07:58:28.168732 398 ts_itest-base.cc:233] Waiting for 1 tablets on tserver d380f310313c4d66a4a7aa4fbf3ecd6c to finish bootstrapping
I20240501 07:58:28.229176 629 heartbeater.cc:503] Master 127.0.99.190:41853 was elected leader, sending a full tablet report...
W20240501 07:58:28.248072 1154 tablet.cc:2369] T 3f5135a82e3742a590ced1958b11cd36 P d380f310313c4d66a4a7aa4fbf3ecd6c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 07:58:28.295096 1023 tablet.cc:2369] T 3f5135a82e3742a590ced1958b11cd36 P 3e83693091294c9da92847195d436d01: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 07:58:28.373051 694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:28.373526 956 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:28.374042 825 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:28.374114 563 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:28.382900 1087 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:58:28.410490 760 heartbeater.cc:503] Master 127.0.99.190:41853 was elected leader, sending a full tablet report...
I20240501 07:58:28.623217 694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:28.624495 563 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:28.624981 956 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:28.625811 825 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:28.629354 1087 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:58:28.690490 1248 consensus_queue.cc:1035] T 3f5135a82e3742a590ced1958b11cd36 P d380f310313c4d66a4a7aa4fbf3ecd6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e83693091294c9da92847195d436d01" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:28.693909 1248 consensus_queue.cc:1035] T 3f5135a82e3742a590ced1958b11cd36 P d380f310313c4d66a4a7aa4fbf3ecd6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "34b0fe1b7b6448fcb991de294f4662d7" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 39243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:28.875523 825 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:28.875720 694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:28.877516 563 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:28.878536 956 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:28.889482 1087 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:58:29.116544 694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:29.117187 563 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:29.117817 825 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:29.118140 956 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:29.121133 1087 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:58:29.213052 956 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:29.213711 563 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:29.214354 1087 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:58:29.221638 694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:29.238517 825 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:29.335557 825 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:29.337132 956 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:29.338057 563 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:29.341914 694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:29.350597 1087 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:58:29.526649 439 ts_manager.cc:292] Set tserver state for 0a922c13fd4443bcb6469c550ced8f76 to MAINTENANCE_MODE
I20240501 07:58:29.741241 563 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:29.741839 825 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:29.742010 694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:29.743333 956 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:29.744076 1087 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:58:29.844731 563 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:29.845765 694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:29.845909 956 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:29.847558 825 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:29.860455 1087 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:58:29.955502 825 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:29.957427 694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:29.959113 563 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:29.967502 956 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:29.981612 1087 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:58:30.059084 563 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:30.059252 694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:30.060143 825 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:30.061066 956 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:30.062156 1087 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:58:30.104732 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tbgEt/build/debug/bin/kudu with pid 632
I20240501 07:58:30.305349 956 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:30.305845 563 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:30.308463 825 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:30.309425 1087 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:58:30.388411 956 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:30.388522 563 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:30.389045 825 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:30.389595 1087 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:58:30.481920 563 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:30.484258 956 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:30.485261 825 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:30.485983 1087 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:58:30.578518 825 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:30.579169 563 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:30.580543 956 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:30.582429 1087 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:58:30.631140 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tbgEt/build/debug/bin/kudu with pid 501
I20240501 07:58:30.637199 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tbgEt/build/debug/bin/kudu with pid 763
I20240501 07:58:30.642933 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tbgEt/build/debug/bin/kudu with pid 894
I20240501 07:58:30.648831 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tbgEt/build/debug/bin/kudu with pid 1025
I20240501 07:58:30.654742 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tbgEt/build/debug/bin/kudu with pid 410
2024-05-01T07:58:30Z chronyd exiting
[ OK ] RebalanceIgnoredTserversTest.Basic (3935 ms)
[----------] 1 test from RebalanceIgnoredTserversTest (3935 ms total)
[----------] 1 test from RebalanceStartCriteriaTest
[ RUN ] RebalanceStartCriteriaTest.TabletServerIsDown/1
I20240501 07:58:30.677543 398 test_util.cc:268] Using random seed: -633885662
I20240501 07:58:30.678339 398 ts_itest-base.cc:112] Starting cluster with:
I20240501 07:58:30.678381 398 ts_itest-base.cc:113] --------------
I20240501 07:58:30.678393 398 ts_itest-base.cc:114] 5 tablet servers
I20240501 07:58:30.678403 398 ts_itest-base.cc:115] 3 replicas per TS
I20240501 07:58:30.678414 398 ts_itest-base.cc:116] --------------
2024-05-01T07:58:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T07:58:30Z Disabled control of system clock
I20240501 07:58:30.686358 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tbgEt/build/debug/bin/kudu
/tmp/dist-test-task7tbgEt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714550306730830-398-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714550306730830-398-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714550306730830-398-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714550306730830-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.99.190:40347
--webserver_interface=127.0.99.190
--webserver_port=0
--builtin_ntp_servers=127.0.99.148:37013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.99.190:40347
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false with env {}
W20240501 07:58:30.795032 1653 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:58:30.795316 1653 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 07:58:30.795392 1653 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:58:30.798584 1653 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 07:58:30.798677 1653 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 07:58:30.798712 1653 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 07:58:30.798744 1653 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 07:58:30.798784 1653 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 07:58:30.802630 1653 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:37013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714550306730830-398-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714550306730830-398-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.0.99.190:40347
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.99.190:40347
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714550306730830-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.0.99.190
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.1653
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714550306730830-398-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 07:43:16 UTC on 1a15035a6830
build id 69
I20240501 07:58:30.803649 1653 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 07:58:30.804854 1653 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:58:30.813083 1658 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:58:30.813767 1661 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:58:30.814460 1659 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:58:30.815651 1653 server_base.cc:1034] running on GCE node
I20240501 07:58:30.816224 1653 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 07:58:30.817271 1653 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 07:58:30.818428 1653 hybrid_clock.cc:648] HybridClock initialized: now 1714550310818415 us; error 30 us; skew 500 ppm
I20240501 07:58:30.820464 1653 webserver.cc:460] Webserver started at http://127.0.99.190:46379/ using document root <none> and password file <none>
I20240501 07:58:30.821101 1653 fs_manager.cc:359] Metadata directory not provided
I20240501 07:58:30.821170 1653 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:58:30.821362 1653 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 07:58:30.823076 1653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714550306730830-398-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "26e1256465524aa39cc3f1d7a15e5bce"
format_stamp: "Formatted at 2024-05-01 07:58:30 on dist-test-slave-40g1"
I20240501 07:58:30.823561 1653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714550306730830-398-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "26e1256465524aa39cc3f1d7a15e5bce"
format_stamp: "Formatted at 2024-05-01 07:58:30 on dist-test-slave-40g1"
I20240501 07:58:30.827360 1653 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240501 07:58:30.829797 1667 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:30.830816 1653 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240501 07:58:30.830957 1653 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714550306730830-398-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714550306730830-398-0/raft_consensus-itest-cluster/master-0/wal
uuid: "26e1256465524aa39cc3f1d7a15e5bce"
format_stamp: "Formatted at 2024-05-01 07:58:30 on dist-test-slave-40g1"
I20240501 07:58:30.831053 1653 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714550306730830-398-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714550306730830-398-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714550306730830-398-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:58:30.844251 1653 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:58:30.844908 1653 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 07:58:30.845099 1653 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:58:30.851737 1653 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.190:40347
I20240501 07:58:30.851754 1719 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.190:40347 every 8 connection(s)
I20240501 07:58:30.852836 1653 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714550306730830-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240501 07:58:30.855513 1720 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:30.860806 1720 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26e1256465524aa39cc3f1d7a15e5bce: Bootstrap starting.
I20240501 07:58:30.861913 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tbgEt/build/debug/bin/kudu as pid 1653
I20240501 07:58:30.862027 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714550306730830-398-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240501 07:58:30.863292 1720 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 26e1256465524aa39cc3f1d7a15e5bce: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:30.864077 1720 log.cc:824] T 00000000000000000000000000000000 P 26e1256465524aa39cc3f1d7a15e5bce: Log is configured to *not* fsync() on all Append() calls
I20240501 07:58:30.865923 1720 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26e1256465524aa39cc3f1d7a15e5bce: No bootstrap required, opened a new log
I20240501 07:58:30.868665 1720 raft_consensus.cc:348] T 00000000000000000000000000000000 P 26e1256465524aa39cc3f1d7a15e5bce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26e1256465524aa39cc3f1d7a15e5bce" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 40347 } }
I20240501 07:58:30.868860 1720 raft_consensus.cc:374] T 00000000000000000000000000000000 P 26e1256465524aa39cc3f1d7a15e5bce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:30.868928 1720 raft_consensus.cc:729] T 00000000000000000000000000000000 P 26e1256465524aa39cc3f1d7a15e5bce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26e1256465524aa39cc3f1d7a15e5bce, State: Initialized, Role: FOLLOWER
I20240501 07:58:30.869418 1720 consensus_queue.cc:260] T 00000000000000000000000000000000 P 26e1256465524aa39cc3f1d7a15e5bce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26e1256465524aa39cc3f1d7a15e5bce" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 40347 } }
I20240501 07:58:30.869592 1720 raft_consensus.cc:388] T 00000000000000000000000000000000 P 26e1256465524aa39cc3f1d7a15e5bce [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 07:58:30.869668 1720 raft_consensus.cc:482] T 00000000000000000000000000000000 P 26e1256465524aa39cc3f1d7a15e5bce [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 07:58:30.869794 1720 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 26e1256465524aa39cc3f1d7a15e5bce [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:30.870759 1720 raft_consensus.cc:504] T 00000000000000000000000000000000 P 26e1256465524aa39cc3f1d7a15e5bce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26e1256465524aa39cc3f1d7a15e5bce" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 40347 } }
I20240501 07:58:30.871086 1720 leader_election.cc:302] T 00000000000000000000000000000000 P 26e1256465524aa39cc3f1d7a15e5bce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 26e1256465524aa39cc3f1d7a15e5bce; no voters:
I20240501 07:58:30.871359 1720 leader_election.cc:288] T 00000000000000000000000000000000 P 26e1256465524aa39cc3f1d7a15e5bce [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 07:58:30.871454 1725 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 26e1256465524aa39cc3f1d7a15e5bce [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:30.871610 1725 raft_consensus.cc:686] T 00000000000000000000000000000000 P 26e1256465524aa39cc3f1d7a15e5bce [term 1 LEADER]: Becoming Leader. State: Replica: 26e1256465524aa39cc3f1d7a15e5bce, State: Running, Role: LEADER
I20240501 07:58:30.871886 1725 consensus_queue.cc:237] T 00000000000000000000000000000000 P 26e1256465524aa39cc3f1d7a15e5bce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26e1256465524aa39cc3f1d7a15e5bce" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 40347 } }
I20240501 07:58:30.872515 1720 sys_catalog.cc:564] T 00000000000000000000000000000000 P 26e1256465524aa39cc3f1d7a15e5bce [sys.catalog]: configured and running, proceeding with master startup.
I20240501 07:58:30.873706 1727 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26e1256465524aa39cc3f1d7a15e5bce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "26e1256465524aa39cc3f1d7a15e5bce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26e1256465524aa39cc3f1d7a15e5bce" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 40347 } } }
I20240501 07:58:30.873766 1726 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26e1256465524aa39cc3f1d7a15e5bce [sys.catalog]: SysCatalogTable state changed. Reason: New leader 26e1256465524aa39cc3f1d7a15e5bce. Latest consensus state: current_term: 1 leader_uuid: "26e1256465524aa39cc3f1d7a15e5bce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26e1256465524aa39cc3f1d7a15e5bce" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 40347 } } }
I20240501 07:58:30.873842 1726 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26e1256465524aa39cc3f1d7a15e5bce [sys.catalog]: This master's current role is: LEADER
I20240501 07:58:30.873842 1727 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26e1256465524aa39cc3f1d7a15e5bce [sys.catalog]: This master's current role is: LEADER
W20240501 07:58:30.877671 1740 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 26e1256465524aa39cc3f1d7a15e5bce: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240501 07:58:30.877758 1740 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240501 07:58:30.877851 1734 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 07:58:30.878583 1734 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 07:58:30.883354 1734 catalog_manager.cc:1347] Generated new cluster ID: 9c30fc5c66d44d26a770ba3b468e77d7
I20240501 07:58:30.883435 1734 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 07:58:30.895027 1734 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 07:58:30.896009 1734 catalog_manager.cc:1504] Loading token signing keys...
I20240501 07:58:30.904726 1734 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 26e1256465524aa39cc3f1d7a15e5bce: Generated new TSK 0
I20240501 07:58:30.905541 1734 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 07:58:30.915324 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tbgEt/build/debug/bin/kudu
/tmp/dist-test-task7tbgEt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.129:0
--local_ip_for_outbound_sockets=127.0.99.129
--webserver_interface=127.0.99.129
--webserver_port=0
--tserver_master_addrs=127.0.99.190:40347
--builtin_ntp_servers=127.0.99.148:37013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240501 07:58:31.029737 1744 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:58:31.029991 1744 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 07:58:31.030040 1744 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:58:31.033305 1744 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 07:58:31.033383 1744 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 07:58:31.033478 1744 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.129
I20240501 07:58:31.037475 1744 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:37013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.0.99.129
--webserver_port=0
--tserver_master_addrs=127.0.99.190:40347
--never_fsync=true
--heap_profile_path=/tmp/kudu.1744
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.129
--log_dir=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 07:43:16 UTC on 1a15035a6830
build id 69
I20240501 07:58:31.038491 1744 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 07:58:31.039680 1744 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:58:31.048944 1752 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:58:31.049641 1750 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:58:31.050694 1749 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:58:31.051543 1744 server_base.cc:1034] running on GCE node
I20240501 07:58:31.051999 1744 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 07:58:31.052629 1744 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 07:58:31.053772 1744 hybrid_clock.cc:648] HybridClock initialized: now 1714550311053759 us; error 33 us; skew 500 ppm
I20240501 07:58:31.055874 1744 webserver.cc:460] Webserver started at http://127.0.99.129:35489/ using document root <none> and password file <none>
I20240501 07:58:31.056447 1744 fs_manager.cc:359] Metadata directory not provided
I20240501 07:58:31.056509 1744 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:58:31.056708 1744 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 07:58:31.058531 1744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "5c60b437192b426092c5d86addeb0fcc"
format_stamp: "Formatted at 2024-05-01 07:58:31 on dist-test-slave-40g1"
I20240501 07:58:31.059041 1744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "5c60b437192b426092c5d86addeb0fcc"
format_stamp: "Formatted at 2024-05-01 07:58:31 on dist-test-slave-40g1"
I20240501 07:58:31.062669 1744 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.005s
I20240501 07:58:31.065301 1758 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:31.066383 1744 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240501 07:58:31.066486 1744 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "5c60b437192b426092c5d86addeb0fcc"
format_stamp: "Formatted at 2024-05-01 07:58:31 on dist-test-slave-40g1"
I20240501 07:58:31.066571 1744 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:58:31.093078 1744 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:58:31.093765 1744 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 07:58:31.093946 1744 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:58:31.094503 1744 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 07:58:31.095429 1744 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 07:58:31.095477 1744 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:31.095518 1744 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 07:58:31.095533 1744 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:31.104210 1744 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.129:37587
I20240501 07:58:31.104243 1871 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.129:37587 every 8 connection(s)
I20240501 07:58:31.105300 1744 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240501 07:58:31.111769 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tbgEt/build/debug/bin/kudu as pid 1744
I20240501 07:58:31.111871 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240501 07:58:31.114171 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tbgEt/build/debug/bin/kudu
/tmp/dist-test-task7tbgEt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.130:0
--local_ip_for_outbound_sockets=127.0.99.130
--webserver_interface=127.0.99.130
--webserver_port=0
--tserver_master_addrs=127.0.99.190:40347
--builtin_ntp_servers=127.0.99.148:37013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 07:58:31.115207 1872 heartbeater.cc:347] Connected to a master server at 127.0.99.190:40347
I20240501 07:58:31.115514 1872 heartbeater.cc:464] Registering TS with master...
I20240501 07:58:31.116294 1872 heartbeater.cc:511] Master 127.0.99.190:40347 requested a full tablet report, sending...
I20240501 07:58:31.118044 1684 ts_manager.cc:194] Registered new tserver with Master: 5c60b437192b426092c5d86addeb0fcc (127.0.99.129:37587)
I20240501 07:58:31.119313 1684 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.129:52591
W20240501 07:58:31.229918 1875 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:58:31.230188 1875 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 07:58:31.230271 1875 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:58:31.234755 1875 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 07:58:31.234869 1875 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 07:58:31.234974 1875 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.130
I20240501 07:58:31.239014 1875 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:37013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.0.99.130
--webserver_port=0
--tserver_master_addrs=127.0.99.190:40347
--never_fsync=true
--heap_profile_path=/tmp/kudu.1875
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.130
--log_dir=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 07:43:16 UTC on 1a15035a6830
build id 69
I20240501 07:58:31.240008 1875 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 07:58:31.241483 1875 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:58:31.250960 1883 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:58:31.251534 1880 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:58:31.252351 1881 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:58:31.253499 1875 server_base.cc:1034] running on GCE node
I20240501 07:58:31.254014 1875 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 07:58:31.254639 1875 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 07:58:31.255828 1875 hybrid_clock.cc:648] HybridClock initialized: now 1714550311255803 us; error 40 us; skew 500 ppm
I20240501 07:58:31.258060 1875 webserver.cc:460] Webserver started at http://127.0.99.130:46729/ using document root <none> and password file <none>
I20240501 07:58:31.258648 1875 fs_manager.cc:359] Metadata directory not provided
I20240501 07:58:31.258713 1875 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:58:31.258900 1875 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 07:58:31.260740 1875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "58c16ef54708456d9f3579096d1b684d"
format_stamp: "Formatted at 2024-05-01 07:58:31 on dist-test-slave-40g1"
I20240501 07:58:31.261281 1875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "58c16ef54708456d9f3579096d1b684d"
format_stamp: "Formatted at 2024-05-01 07:58:31 on dist-test-slave-40g1"
I20240501 07:58:31.264818 1875 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240501 07:58:31.267319 1889 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:31.268409 1875 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 07:58:31.268539 1875 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "58c16ef54708456d9f3579096d1b684d"
format_stamp: "Formatted at 2024-05-01 07:58:31 on dist-test-slave-40g1"
I20240501 07:58:31.268649 1875 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:58:31.286060 1875 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:58:31.286726 1875 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 07:58:31.286899 1875 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:58:31.287436 1875 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 07:58:31.288394 1875 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 07:58:31.288442 1875 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:31.288504 1875 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 07:58:31.288530 1875 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:31.297272 1875 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.130:41127
I20240501 07:58:31.297294 2002 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.130:41127 every 8 connection(s)
I20240501 07:58:31.298301 1875 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240501 07:58:31.301100 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tbgEt/build/debug/bin/kudu as pid 1875
I20240501 07:58:31.301224 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240501 07:58:31.303692 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tbgEt/build/debug/bin/kudu
/tmp/dist-test-task7tbgEt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.131:0
--local_ip_for_outbound_sockets=127.0.99.131
--webserver_interface=127.0.99.131
--webserver_port=0
--tserver_master_addrs=127.0.99.190:40347
--builtin_ntp_servers=127.0.99.148:37013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 07:58:31.308161 2003 heartbeater.cc:347] Connected to a master server at 127.0.99.190:40347
I20240501 07:58:31.308422 2003 heartbeater.cc:464] Registering TS with master...
I20240501 07:58:31.309073 2003 heartbeater.cc:511] Master 127.0.99.190:40347 requested a full tablet report, sending...
I20240501 07:58:31.310250 1684 ts_manager.cc:194] Registered new tserver with Master: 58c16ef54708456d9f3579096d1b684d (127.0.99.130:41127)
I20240501 07:58:31.311139 1684 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.130:35109
W20240501 07:58:31.417049 2006 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:58:31.417296 2006 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 07:58:31.417346 2006 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:58:31.420486 2006 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 07:58:31.420661 2006 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 07:58:31.420785 2006 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.131
I20240501 07:58:31.424621 2006 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:37013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.0.99.131
--webserver_port=0
--tserver_master_addrs=127.0.99.190:40347
--never_fsync=true
--heap_profile_path=/tmp/kudu.2006
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.131
--log_dir=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 07:43:16 UTC on 1a15035a6830
build id 69
I20240501 07:58:31.425573 2006 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 07:58:31.426693 2006 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:58:31.435626 2011 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:58:31.436568 2012 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:58:31.437326 2014 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:58:31.438130 2006 server_base.cc:1034] running on GCE node
I20240501 07:58:31.438549 2006 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 07:58:31.439128 2006 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 07:58:31.440363 2006 hybrid_clock.cc:648] HybridClock initialized: now 1714550311440333 us; error 42 us; skew 500 ppm
I20240501 07:58:31.442458 2006 webserver.cc:460] Webserver started at http://127.0.99.131:33601/ using document root <none> and password file <none>
I20240501 07:58:31.443069 2006 fs_manager.cc:359] Metadata directory not provided
I20240501 07:58:31.443132 2006 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:58:31.443318 2006 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 07:58:31.445063 2006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "e75a8b0b09784a38890dcdca9c1df009"
format_stamp: "Formatted at 2024-05-01 07:58:31 on dist-test-slave-40g1"
I20240501 07:58:31.445572 2006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "e75a8b0b09784a38890dcdca9c1df009"
format_stamp: "Formatted at 2024-05-01 07:58:31 on dist-test-slave-40g1"
I20240501 07:58:31.449216 2006 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240501 07:58:31.451454 2020 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:31.452541 2006 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 07:58:31.452661 2006 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "e75a8b0b09784a38890dcdca9c1df009"
format_stamp: "Formatted at 2024-05-01 07:58:31 on dist-test-slave-40g1"
I20240501 07:58:31.452764 2006 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:58:31.469111 2006 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:58:31.469823 2006 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 07:58:31.469988 2006 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:58:31.470590 2006 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 07:58:31.471524 2006 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 07:58:31.471570 2006 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:31.471611 2006 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 07:58:31.471634 2006 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:31.480309 2006 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.131:40823
I20240501 07:58:31.480341 2133 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.131:40823 every 8 connection(s)
I20240501 07:58:31.481335 2006 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240501 07:58:31.490023 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tbgEt/build/debug/bin/kudu as pid 2006
I20240501 07:58:31.490149 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240501 07:58:31.490507 2134 heartbeater.cc:347] Connected to a master server at 127.0.99.190:40347
I20240501 07:58:31.490782 2134 heartbeater.cc:464] Registering TS with master...
I20240501 07:58:31.491438 2134 heartbeater.cc:511] Master 127.0.99.190:40347 requested a full tablet report, sending...
I20240501 07:58:31.492213 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tbgEt/build/debug/bin/kudu
/tmp/dist-test-task7tbgEt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.132:0
--local_ip_for_outbound_sockets=127.0.99.132
--webserver_interface=127.0.99.132
--webserver_port=0
--tserver_master_addrs=127.0.99.190:40347
--builtin_ntp_servers=127.0.99.148:37013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 07:58:31.492476 1684 ts_manager.cc:194] Registered new tserver with Master: e75a8b0b09784a38890dcdca9c1df009 (127.0.99.131:40823)
I20240501 07:58:31.493265 1684 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.131:59315
W20240501 07:58:31.604717 2137 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:58:31.605003 2137 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 07:58:31.605055 2137 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:58:31.608228 2137 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 07:58:31.608305 2137 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 07:58:31.608381 2137 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.132
I20240501 07:58:31.613379 2137 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:37013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.0.99.132
--webserver_port=0
--tserver_master_addrs=127.0.99.190:40347
--never_fsync=true
--heap_profile_path=/tmp/kudu.2137
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.132
--log_dir=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 07:43:16 UTC on 1a15035a6830
build id 69
I20240501 07:58:31.614439 2137 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 07:58:31.615622 2137 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:58:31.624750 2142 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:58:31.625361 2145 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:58:31.626085 2143 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:58:31.627293 2137 server_base.cc:1034] running on GCE node
I20240501 07:58:31.627789 2137 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 07:58:31.628415 2137 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 07:58:31.629568 2137 hybrid_clock.cc:648] HybridClock initialized: now 1714550311629548 us; error 32 us; skew 500 ppm
I20240501 07:58:31.631532 2137 webserver.cc:460] Webserver started at http://127.0.99.132:41129/ using document root <none> and password file <none>
I20240501 07:58:31.632122 2137 fs_manager.cc:359] Metadata directory not provided
I20240501 07:58:31.632180 2137 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:58:31.632355 2137 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 07:58:31.634094 2137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "f499f33a65a847b69ad7ac12bb8acc5f"
format_stamp: "Formatted at 2024-05-01 07:58:31 on dist-test-slave-40g1"
I20240501 07:58:31.634595 2137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "f499f33a65a847b69ad7ac12bb8acc5f"
format_stamp: "Formatted at 2024-05-01 07:58:31 on dist-test-slave-40g1"
I20240501 07:58:31.638175 2137 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240501 07:58:31.640373 2151 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:31.641546 2137 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240501 07:58:31.641719 2137 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "f499f33a65a847b69ad7ac12bb8acc5f"
format_stamp: "Formatted at 2024-05-01 07:58:31 on dist-test-slave-40g1"
I20240501 07:58:31.641824 2137 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:58:31.681293 2137 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:58:31.681973 2137 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 07:58:31.682142 2137 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:58:31.682686 2137 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 07:58:31.683575 2137 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 07:58:31.683622 2137 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:31.683682 2137 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 07:58:31.683709 2137 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:31.692054 2137 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.132:43327
I20240501 07:58:31.692082 2264 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.132:43327 every 8 connection(s)
I20240501 07:58:31.693004 2137 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240501 07:58:31.699206 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tbgEt/build/debug/bin/kudu as pid 2137
I20240501 07:58:31.699329 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240501 07:58:31.701517 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tbgEt/build/debug/bin/kudu
/tmp/dist-test-task7tbgEt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.133:0
--local_ip_for_outbound_sockets=127.0.99.133
--webserver_interface=127.0.99.133
--webserver_port=0
--tserver_master_addrs=127.0.99.190:40347
--builtin_ntp_servers=127.0.99.148:37013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 07:58:31.701728 2265 heartbeater.cc:347] Connected to a master server at 127.0.99.190:40347
I20240501 07:58:31.701936 2265 heartbeater.cc:464] Registering TS with master...
I20240501 07:58:31.702446 2265 heartbeater.cc:511] Master 127.0.99.190:40347 requested a full tablet report, sending...
I20240501 07:58:31.703415 1684 ts_manager.cc:194] Registered new tserver with Master: f499f33a65a847b69ad7ac12bb8acc5f (127.0.99.132:43327)
I20240501 07:58:31.703959 1684 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.132:56365
W20240501 07:58:31.812835 2268 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:58:31.813129 2268 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 07:58:31.813213 2268 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:58:31.816347 2268 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 07:58:31.816429 2268 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 07:58:31.816533 2268 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.133
I20240501 07:58:31.820485 2268 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:37013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.133:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.0.99.133
--webserver_port=0
--tserver_master_addrs=127.0.99.190:40347
--never_fsync=true
--heap_profile_path=/tmp/kudu.2268
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.133
--log_dir=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 07:43:16 UTC on 1a15035a6830
build id 69
I20240501 07:58:31.821667 2268 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 07:58:31.822865 2268 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:58:31.832047 2274 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:58:31.832933 2276 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:58:31.833813 2273 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:58:31.834681 2268 server_base.cc:1034] running on GCE node
I20240501 07:58:31.835156 2268 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 07:58:31.835839 2268 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 07:58:31.837029 2268 hybrid_clock.cc:648] HybridClock initialized: now 1714550311837007 us; error 35 us; skew 500 ppm
I20240501 07:58:31.839080 2268 webserver.cc:460] Webserver started at http://127.0.99.133:38553/ using document root <none> and password file <none>
I20240501 07:58:31.839640 2268 fs_manager.cc:359] Metadata directory not provided
I20240501 07:58:31.839697 2268 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:58:31.839869 2268 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 07:58:31.841621 2268 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "591dca46823d41688d41d4a717ee77aa"
format_stamp: "Formatted at 2024-05-01 07:58:31 on dist-test-slave-40g1"
I20240501 07:58:31.842137 2268 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "591dca46823d41688d41d4a717ee77aa"
format_stamp: "Formatted at 2024-05-01 07:58:31 on dist-test-slave-40g1"
I20240501 07:58:31.845718 2268 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20240501 07:58:31.848147 2282 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:31.849227 2268 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20240501 07:58:31.849331 2268 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "591dca46823d41688d41d4a717ee77aa"
format_stamp: "Formatted at 2024-05-01 07:58:31 on dist-test-slave-40g1"
I20240501 07:58:31.849416 2268 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:58:31.866034 2268 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:58:31.866732 2268 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 07:58:31.866886 2268 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:58:31.867452 2268 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 07:58:31.868389 2268 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 07:58:31.868438 2268 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:31.868479 2268 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 07:58:31.868494 2268 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:31.877460 2395 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.133:43601 every 8 connection(s)
I20240501 07:58:31.877429 2268 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.133:43601
I20240501 07:58:31.878608 2268 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 07:58:31.887302 2396 heartbeater.cc:347] Connected to a master server at 127.0.99.190:40347
I20240501 07:58:31.887502 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tbgEt/build/debug/bin/kudu as pid 2268
I20240501 07:58:31.887547 2396 heartbeater.cc:464] Registering TS with master...
I20240501 07:58:31.887607 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240501 07:58:31.888087 2396 heartbeater.cc:511] Master 127.0.99.190:40347 requested a full tablet report, sending...
I20240501 07:58:31.889127 1684 ts_manager.cc:194] Registered new tserver with Master: 591dca46823d41688d41d4a717ee77aa (127.0.99.133:43601)
I20240501 07:58:31.889667 1684 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.133:53599
I20240501 07:58:31.889811 398 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240501 07:58:31.898756 1684 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46800:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240501 07:58:31.916810 2068 tablet_service.cc:1467] Processing CreateTablet for tablet 9ed4fc819e6d48be916b7fa53344bc60 (DEFAULT_TABLE table=TestTable [id=1aa3cafe340f47e1aa0ee749753304e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:31.917038 2330 tablet_service.cc:1467] Processing CreateTablet for tablet 9ed4fc819e6d48be916b7fa53344bc60 (DEFAULT_TABLE table=TestTable [id=1aa3cafe340f47e1aa0ee749753304e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:31.917038 1937 tablet_service.cc:1467] Processing CreateTablet for tablet 9ed4fc819e6d48be916b7fa53344bc60 (DEFAULT_TABLE table=TestTable [id=1aa3cafe340f47e1aa0ee749753304e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:31.917795 2068 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ed4fc819e6d48be916b7fa53344bc60. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:31.917918 1937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ed4fc819e6d48be916b7fa53344bc60. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:31.917918 2330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ed4fc819e6d48be916b7fa53344bc60. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:31.922802 2413 tablet_bootstrap.cc:492] T 9ed4fc819e6d48be916b7fa53344bc60 P e75a8b0b09784a38890dcdca9c1df009: Bootstrap starting.
I20240501 07:58:31.924302 2414 tablet_bootstrap.cc:492] T 9ed4fc819e6d48be916b7fa53344bc60 P 58c16ef54708456d9f3579096d1b684d: Bootstrap starting.
I20240501 07:58:31.924811 2415 tablet_bootstrap.cc:492] T 9ed4fc819e6d48be916b7fa53344bc60 P 591dca46823d41688d41d4a717ee77aa: Bootstrap starting.
I20240501 07:58:31.925130 2413 tablet_bootstrap.cc:654] T 9ed4fc819e6d48be916b7fa53344bc60 P e75a8b0b09784a38890dcdca9c1df009: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:31.925946 2413 log.cc:824] T 9ed4fc819e6d48be916b7fa53344bc60 P e75a8b0b09784a38890dcdca9c1df009: Log is configured to *not* fsync() on all Append() calls
I20240501 07:58:31.926896 2415 tablet_bootstrap.cc:654] T 9ed4fc819e6d48be916b7fa53344bc60 P 591dca46823d41688d41d4a717ee77aa: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:31.927664 2415 log.cc:824] T 9ed4fc819e6d48be916b7fa53344bc60 P 591dca46823d41688d41d4a717ee77aa: Log is configured to *not* fsync() on all Append() calls
I20240501 07:58:31.927699 2413 tablet_bootstrap.cc:492] T 9ed4fc819e6d48be916b7fa53344bc60 P e75a8b0b09784a38890dcdca9c1df009: No bootstrap required, opened a new log
I20240501 07:58:31.927959 2413 ts_tablet_manager.cc:1367] T 9ed4fc819e6d48be916b7fa53344bc60 P e75a8b0b09784a38890dcdca9c1df009: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.005s
I20240501 07:58:31.928313 2414 tablet_bootstrap.cc:654] T 9ed4fc819e6d48be916b7fa53344bc60 P 58c16ef54708456d9f3579096d1b684d: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:31.929144 2414 log.cc:824] T 9ed4fc819e6d48be916b7fa53344bc60 P 58c16ef54708456d9f3579096d1b684d: Log is configured to *not* fsync() on all Append() calls
I20240501 07:58:31.930078 2415 tablet_bootstrap.cc:492] T 9ed4fc819e6d48be916b7fa53344bc60 P 591dca46823d41688d41d4a717ee77aa: No bootstrap required, opened a new log
I20240501 07:58:31.930295 2415 ts_tablet_manager.cc:1367] T 9ed4fc819e6d48be916b7fa53344bc60 P 591dca46823d41688d41d4a717ee77aa: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240501 07:58:31.930519 2414 tablet_bootstrap.cc:492] T 9ed4fc819e6d48be916b7fa53344bc60 P 58c16ef54708456d9f3579096d1b684d: No bootstrap required, opened a new log
I20240501 07:58:31.930636 2414 ts_tablet_manager.cc:1367] T 9ed4fc819e6d48be916b7fa53344bc60 P 58c16ef54708456d9f3579096d1b684d: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20240501 07:58:31.931893 2413 raft_consensus.cc:348] T 9ed4fc819e6d48be916b7fa53344bc60 P e75a8b0b09784a38890dcdca9c1df009 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e75a8b0b09784a38890dcdca9c1df009" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40823 } } peers { permanent_uuid: "591dca46823d41688d41d4a717ee77aa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 43601 } } peers { permanent_uuid: "58c16ef54708456d9f3579096d1b684d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41127 } }
I20240501 07:58:31.932116 2413 raft_consensus.cc:374] T 9ed4fc819e6d48be916b7fa53344bc60 P e75a8b0b09784a38890dcdca9c1df009 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:31.932166 2413 raft_consensus.cc:729] T 9ed4fc819e6d48be916b7fa53344bc60 P e75a8b0b09784a38890dcdca9c1df009 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e75a8b0b09784a38890dcdca9c1df009, State: Initialized, Role: FOLLOWER
I20240501 07:58:31.932783 2413 consensus_queue.cc:260] T 9ed4fc819e6d48be916b7fa53344bc60 P e75a8b0b09784a38890dcdca9c1df009 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e75a8b0b09784a38890dcdca9c1df009" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40823 } } peers { permanent_uuid: "591dca46823d41688d41d4a717ee77aa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 43601 } } peers { permanent_uuid: "58c16ef54708456d9f3579096d1b684d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41127 } }
I20240501 07:58:31.933053 2414 raft_consensus.cc:348] T 9ed4fc819e6d48be916b7fa53344bc60 P 58c16ef54708456d9f3579096d1b684d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e75a8b0b09784a38890dcdca9c1df009" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40823 } } peers { permanent_uuid: "591dca46823d41688d41d4a717ee77aa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 43601 } } peers { permanent_uuid: "58c16ef54708456d9f3579096d1b684d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41127 } }
I20240501 07:58:31.933293 2414 raft_consensus.cc:374] T 9ed4fc819e6d48be916b7fa53344bc60 P 58c16ef54708456d9f3579096d1b684d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:31.933372 2414 raft_consensus.cc:729] T 9ed4fc819e6d48be916b7fa53344bc60 P 58c16ef54708456d9f3579096d1b684d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58c16ef54708456d9f3579096d1b684d, State: Initialized, Role: FOLLOWER
I20240501 07:58:31.933854 2413 ts_tablet_manager.cc:1398] T 9ed4fc819e6d48be916b7fa53344bc60 P e75a8b0b09784a38890dcdca9c1df009: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
I20240501 07:58:31.933861 2414 consensus_queue.cc:260] T 9ed4fc819e6d48be916b7fa53344bc60 P 58c16ef54708456d9f3579096d1b684d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e75a8b0b09784a38890dcdca9c1df009" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40823 } } peers { permanent_uuid: "591dca46823d41688d41d4a717ee77aa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 43601 } } peers { permanent_uuid: "58c16ef54708456d9f3579096d1b684d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41127 } }
I20240501 07:58:31.934355 2134 heartbeater.cc:503] Master 127.0.99.190:40347 was elected leader, sending a full tablet report...
I20240501 07:58:31.934443 2415 raft_consensus.cc:348] T 9ed4fc819e6d48be916b7fa53344bc60 P 591dca46823d41688d41d4a717ee77aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e75a8b0b09784a38890dcdca9c1df009" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40823 } } peers { permanent_uuid: "591dca46823d41688d41d4a717ee77aa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 43601 } } peers { permanent_uuid: "58c16ef54708456d9f3579096d1b684d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41127 } }
I20240501 07:58:31.934621 2415 raft_consensus.cc:374] T 9ed4fc819e6d48be916b7fa53344bc60 P 591dca46823d41688d41d4a717ee77aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:31.934671 2415 raft_consensus.cc:729] T 9ed4fc819e6d48be916b7fa53344bc60 P 591dca46823d41688d41d4a717ee77aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 591dca46823d41688d41d4a717ee77aa, State: Initialized, Role: FOLLOWER
I20240501 07:58:31.934676 2414 ts_tablet_manager.cc:1398] T 9ed4fc819e6d48be916b7fa53344bc60 P 58c16ef54708456d9f3579096d1b684d: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20240501 07:58:31.935007 2003 heartbeater.cc:503] Master 127.0.99.190:40347 was elected leader, sending a full tablet report...
I20240501 07:58:31.935062 2415 consensus_queue.cc:260] T 9ed4fc819e6d48be916b7fa53344bc60 P 591dca46823d41688d41d4a717ee77aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e75a8b0b09784a38890dcdca9c1df009" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40823 } } peers { permanent_uuid: "591dca46823d41688d41d4a717ee77aa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 43601 } } peers { permanent_uuid: "58c16ef54708456d9f3579096d1b684d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41127 } }
I20240501 07:58:31.935899 2415 ts_tablet_manager.cc:1398] T 9ed4fc819e6d48be916b7fa53344bc60 P 591dca46823d41688d41d4a717ee77aa: Time spent starting tablet: real 0.006s user 0.002s sys 0.003s
I20240501 07:58:31.936239 2396 heartbeater.cc:503] Master 127.0.99.190:40347 was elected leader, sending a full tablet report...
I20240501 07:58:31.973145 2421 raft_consensus.cc:482] T 9ed4fc819e6d48be916b7fa53344bc60 P 591dca46823d41688d41d4a717ee77aa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:31.973349 2421 raft_consensus.cc:504] T 9ed4fc819e6d48be916b7fa53344bc60 P 591dca46823d41688d41d4a717ee77aa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e75a8b0b09784a38890dcdca9c1df009" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40823 } } peers { permanent_uuid: "591dca46823d41688d41d4a717ee77aa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 43601 } } peers { permanent_uuid: "58c16ef54708456d9f3579096d1b684d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41127 } }
I20240501 07:58:31.974524 2421 leader_election.cc:288] T 9ed4fc819e6d48be916b7fa53344bc60 P 591dca46823d41688d41d4a717ee77aa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e75a8b0b09784a38890dcdca9c1df009 (127.0.99.131:40823), 58c16ef54708456d9f3579096d1b684d (127.0.99.130:41127)
I20240501 07:58:31.979009 1957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ed4fc819e6d48be916b7fa53344bc60" candidate_uuid: "591dca46823d41688d41d4a717ee77aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58c16ef54708456d9f3579096d1b684d" is_pre_election: true
I20240501 07:58:31.979080 2088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ed4fc819e6d48be916b7fa53344bc60" candidate_uuid: "591dca46823d41688d41d4a717ee77aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e75a8b0b09784a38890dcdca9c1df009" is_pre_election: true
I20240501 07:58:31.979321 1957 raft_consensus.cc:2454] T 9ed4fc819e6d48be916b7fa53344bc60 P 58c16ef54708456d9f3579096d1b684d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 591dca46823d41688d41d4a717ee77aa in term 0.
I20240501 07:58:31.979347 2088 raft_consensus.cc:2454] T 9ed4fc819e6d48be916b7fa53344bc60 P e75a8b0b09784a38890dcdca9c1df009 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 591dca46823d41688d41d4a717ee77aa in term 0.
I20240501 07:58:31.979748 2283 leader_election.cc:302] T 9ed4fc819e6d48be916b7fa53344bc60 P 591dca46823d41688d41d4a717ee77aa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58c16ef54708456d9f3579096d1b684d, 591dca46823d41688d41d4a717ee77aa; no voters:
I20240501 07:58:31.980000 2421 raft_consensus.cc:2781] T 9ed4fc819e6d48be916b7fa53344bc60 P 591dca46823d41688d41d4a717ee77aa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:31.980072 2421 raft_consensus.cc:482] T 9ed4fc819e6d48be916b7fa53344bc60 P 591dca46823d41688d41d4a717ee77aa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:31.980103 2421 raft_consensus.cc:3037] T 9ed4fc819e6d48be916b7fa53344bc60 P 591dca46823d41688d41d4a717ee77aa [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:31.981046 2421 raft_consensus.cc:504] T 9ed4fc819e6d48be916b7fa53344bc60 P 591dca46823d41688d41d4a717ee77aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e75a8b0b09784a38890dcdca9c1df009" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40823 } } peers { permanent_uuid: "591dca46823d41688d41d4a717ee77aa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 43601 } } peers { permanent_uuid: "58c16ef54708456d9f3579096d1b684d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41127 } }
I20240501 07:58:31.981418 2421 leader_election.cc:288] T 9ed4fc819e6d48be916b7fa53344bc60 P 591dca46823d41688d41d4a717ee77aa [CANDIDATE]: Term 1 election: Requested vote from peers e75a8b0b09784a38890dcdca9c1df009 (127.0.99.131:40823), 58c16ef54708456d9f3579096d1b684d (127.0.99.130:41127)
I20240501 07:58:31.981752 2088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ed4fc819e6d48be916b7fa53344bc60" candidate_uuid: "591dca46823d41688d41d4a717ee77aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e75a8b0b09784a38890dcdca9c1df009"
I20240501 07:58:31.981830 1957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ed4fc819e6d48be916b7fa53344bc60" candidate_uuid: "591dca46823d41688d41d4a717ee77aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58c16ef54708456d9f3579096d1b684d"
I20240501 07:58:31.981869 2088 raft_consensus.cc:3037] T 9ed4fc819e6d48be916b7fa53344bc60 P e75a8b0b09784a38890dcdca9c1df009 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:31.981972 1957 raft_consensus.cc:3037] T 9ed4fc819e6d48be916b7fa53344bc60 P 58c16ef54708456d9f3579096d1b684d [term 0 FOLLOWER]: Advancing to term 1
W20240501 07:58:31.982442 2135 tablet.cc:2369] T 9ed4fc819e6d48be916b7fa53344bc60 P e75a8b0b09784a38890dcdca9c1df009: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 07:58:31.982895 2088 raft_consensus.cc:2454] T 9ed4fc819e6d48be916b7fa53344bc60 P e75a8b0b09784a38890dcdca9c1df009 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 591dca46823d41688d41d4a717ee77aa in term 1.
I20240501 07:58:31.982906 1957 raft_consensus.cc:2454] T 9ed4fc819e6d48be916b7fa53344bc60 P 58c16ef54708456d9f3579096d1b684d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 591dca46823d41688d41d4a717ee77aa in term 1.
I20240501 07:58:31.983199 2286 leader_election.cc:302] T 9ed4fc819e6d48be916b7fa53344bc60 P 591dca46823d41688d41d4a717ee77aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 591dca46823d41688d41d4a717ee77aa, e75a8b0b09784a38890dcdca9c1df009; no voters:
I20240501 07:58:31.983374 2421 raft_consensus.cc:2781] T 9ed4fc819e6d48be916b7fa53344bc60 P 591dca46823d41688d41d4a717ee77aa [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:31.983621 2421 raft_consensus.cc:686] T 9ed4fc819e6d48be916b7fa53344bc60 P 591dca46823d41688d41d4a717ee77aa [term 1 LEADER]: Becoming Leader. State: Replica: 591dca46823d41688d41d4a717ee77aa, State: Running, Role: LEADER
I20240501 07:58:31.983952 2421 consensus_queue.cc:237] T 9ed4fc819e6d48be916b7fa53344bc60 P 591dca46823d41688d41d4a717ee77aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e75a8b0b09784a38890dcdca9c1df009" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40823 } } peers { permanent_uuid: "591dca46823d41688d41d4a717ee77aa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 43601 } } peers { permanent_uuid: "58c16ef54708456d9f3579096d1b684d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41127 } }
I20240501 07:58:31.987013 1683 catalog_manager.cc:5484] T 9ed4fc819e6d48be916b7fa53344bc60 P 591dca46823d41688d41d4a717ee77aa reported cstate change: term changed from 0 to 1, leader changed from <none> to 591dca46823d41688d41d4a717ee77aa (127.0.99.133). New cstate: current_term: 1 leader_uuid: "591dca46823d41688d41d4a717ee77aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e75a8b0b09784a38890dcdca9c1df009" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40823 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "591dca46823d41688d41d4a717ee77aa" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 43601 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58c16ef54708456d9f3579096d1b684d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41127 } health_report { overall_health: UNKNOWN } } }
I20240501 07:58:31.998448 398 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240501 07:58:31.999869 398 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 5c60b437192b426092c5d86addeb0fcc to finish bootstrapping
I20240501 07:58:32.010411 398 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 58c16ef54708456d9f3579096d1b684d to finish bootstrapping
I20240501 07:58:32.015615 398 ts_itest-base.cc:233] Waiting for 1 tablets on tserver e75a8b0b09784a38890dcdca9c1df009 to finish bootstrapping
I20240501 07:58:32.021422 398 ts_itest-base.cc:233] Waiting for 0 tablets on tserver f499f33a65a847b69ad7ac12bb8acc5f to finish bootstrapping
I20240501 07:58:32.028033 398 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 591dca46823d41688d41d4a717ee77aa to finish bootstrapping
I20240501 07:58:32.033037 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tbgEt/build/debug/bin/kudu with pid 1744
W20240501 07:58:32.049762 2004 tablet.cc:2369] T 9ed4fc819e6d48be916b7fa53344bc60 P 58c16ef54708456d9f3579096d1b684d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 07:58:32.129509 2397 tablet.cc:2369] T 9ed4fc819e6d48be916b7fa53344bc60 P 591dca46823d41688d41d4a717ee77aa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 07:58:32.228829 2068 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:32.228828 2330 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:58:32.229149 1937 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:32.229701 2199 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:32.315757 1937 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:32.316362 2330 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:58:32.316576 2068 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:32.316792 2199 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:32.410732 1937 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:32.411195 2330 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:58:32.412218 2068 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:32.418112 2199 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:32.440613 2421 consensus_queue.cc:1035] T 9ed4fc819e6d48be916b7fa53344bc60 P 591dca46823d41688d41d4a717ee77aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "58c16ef54708456d9f3579096d1b684d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:32.447558 2518 consensus_queue.cc:1035] T 9ed4fc819e6d48be916b7fa53344bc60 P 591dca46823d41688d41d4a717ee77aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "e75a8b0b09784a38890dcdca9c1df009" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:32.640257 1937 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:32.641117 2068 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:32.641441 2199 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:32.642220 2330 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:58:32.705871 2265 heartbeater.cc:503] Master 127.0.99.190:40347 was elected leader, sending a full tablet report...
I20240501 07:58:32.721717 1937 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:32.722265 2199 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:32.723193 2330 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:58:32.723450 2068 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:32.819329 1937 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:32.819820 2330 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:58:32.819844 2068 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:32.821118 2199 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:32.853219 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tbgEt/build/debug/bin/kudu with pid 1875
I20240501 07:58:32.860843 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tbgEt/build/debug/bin/kudu with pid 2006
I20240501 07:58:32.867089 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tbgEt/build/debug/bin/kudu with pid 2137
I20240501 07:58:32.873786 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tbgEt/build/debug/bin/kudu with pid 2268
I20240501 07:58:32.880672 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tbgEt/build/debug/bin/kudu with pid 1653
2024-05-01T07:58:32Z chronyd exiting
[ OK ] RebalanceStartCriteriaTest.TabletServerIsDown/1 (2226 ms)
[----------] 1 test from RebalanceStartCriteriaTest (2226 ms total)
[----------] 1 test from RebalanceParamTest
[ RUN ] RebalanceParamTest.Rebalance/1
I20240501 07:58:32.904063 398 test_util.cc:268] Using random seed: -631659143
I20240501 07:58:32.904829 398 ts_itest-base.cc:112] Starting cluster with:
I20240501 07:58:32.904867 398 ts_itest-base.cc:113] --------------
I20240501 07:58:32.904878 398 ts_itest-base.cc:114] 7 tablet servers
I20240501 07:58:32.904888 398 ts_itest-base.cc:115] 1 replicas per TS
I20240501 07:58:32.904898 398 ts_itest-base.cc:116] --------------
2024-05-01T07:58:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T07:58:32Z Disabled control of system clock
I20240501 07:58:32.913007 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tbgEt/build/debug/bin/kudu
/tmp/dist-test-task7tbgEt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.99.190:42355
--webserver_interface=127.0.99.190
--webserver_port=0
--builtin_ntp_servers=127.0.99.148:34309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.99.190:42355
--allow_unsafe_replication_factor
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--tserver_unresponsive_timeout_ms=3000 with env {}
W20240501 07:58:33.023623 2618 flags.cc:425] Enabled unsafe flag: --allow_unsafe_replication_factor=true
W20240501 07:58:33.023924 2618 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:58:33.023980 2618 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 07:58:33.024050 2618 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:58:33.027199 2618 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 07:58:33.027326 2618 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 07:58:33.027362 2618 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 07:58:33.027381 2618 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 07:58:33.027403 2618 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 07:58:33.031261 2618 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:34309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/master-0/wal
--allow_unsafe_replication_factor=true
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.0.99.190:42355
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.99.190:42355
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.0.99.190
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.2618
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 07:43:16 UTC on 1a15035a6830
build id 69
I20240501 07:58:33.032217 2618 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 07:58:33.033362 2618 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:58:33.041539 2623 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:58:33.042264 2624 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:58:33.043284 2626 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:58:33.044302 2618 server_base.cc:1034] running on GCE node
I20240501 07:58:33.044921 2618 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 07:58:33.045930 2618 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 07:58:33.047106 2618 hybrid_clock.cc:648] HybridClock initialized: now 1714550313047077 us; error 45 us; skew 500 ppm
I20240501 07:58:33.049268 2618 webserver.cc:460] Webserver started at http://127.0.99.190:34493/ using document root <none> and password file <none>
I20240501 07:58:33.050029 2618 fs_manager.cc:359] Metadata directory not provided
I20240501 07:58:33.050112 2618 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:58:33.050366 2618 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 07:58:33.052121 2618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "95f14cded5744864a8da22ac2b8b06fa"
format_stamp: "Formatted at 2024-05-01 07:58:33 on dist-test-slave-40g1"
I20240501 07:58:33.052624 2618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "95f14cded5744864a8da22ac2b8b06fa"
format_stamp: "Formatted at 2024-05-01 07:58:33 on dist-test-slave-40g1"
I20240501 07:58:33.056130 2618 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240501 07:58:33.058508 2632 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:33.059533 2618 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240501 07:58:33.059653 2618 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/master-0/wal
uuid: "95f14cded5744864a8da22ac2b8b06fa"
format_stamp: "Formatted at 2024-05-01 07:58:33 on dist-test-slave-40g1"
I20240501 07:58:33.059747 2618 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:58:33.085368 2618 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:58:33.086077 2618 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 07:58:33.086242 2618 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:58:33.093284 2618 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.190:42355
I20240501 07:58:33.093303 2684 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.190:42355 every 8 connection(s)
I20240501 07:58:33.094333 2618 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240501 07:58:33.097023 2685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:33.098594 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tbgEt/build/debug/bin/kudu as pid 2618
I20240501 07:58:33.098702 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240501 07:58:33.103042 2685 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95f14cded5744864a8da22ac2b8b06fa: Bootstrap starting.
I20240501 07:58:33.105659 2685 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 95f14cded5744864a8da22ac2b8b06fa: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:33.106537 2685 log.cc:824] T 00000000000000000000000000000000 P 95f14cded5744864a8da22ac2b8b06fa: Log is configured to *not* fsync() on all Append() calls
I20240501 07:58:33.108328 2685 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95f14cded5744864a8da22ac2b8b06fa: No bootstrap required, opened a new log
I20240501 07:58:33.110971 2685 raft_consensus.cc:348] T 00000000000000000000000000000000 P 95f14cded5744864a8da22ac2b8b06fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95f14cded5744864a8da22ac2b8b06fa" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 42355 } }
I20240501 07:58:33.111173 2685 raft_consensus.cc:374] T 00000000000000000000000000000000 P 95f14cded5744864a8da22ac2b8b06fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:33.111225 2685 raft_consensus.cc:729] T 00000000000000000000000000000000 P 95f14cded5744864a8da22ac2b8b06fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95f14cded5744864a8da22ac2b8b06fa, State: Initialized, Role: FOLLOWER
I20240501 07:58:33.111634 2685 consensus_queue.cc:260] T 00000000000000000000000000000000 P 95f14cded5744864a8da22ac2b8b06fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95f14cded5744864a8da22ac2b8b06fa" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 42355 } }
I20240501 07:58:33.111749 2685 raft_consensus.cc:388] T 00000000000000000000000000000000 P 95f14cded5744864a8da22ac2b8b06fa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 07:58:33.111811 2685 raft_consensus.cc:482] T 00000000000000000000000000000000 P 95f14cded5744864a8da22ac2b8b06fa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 07:58:33.111912 2685 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 95f14cded5744864a8da22ac2b8b06fa [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:33.112689 2685 raft_consensus.cc:504] T 00000000000000000000000000000000 P 95f14cded5744864a8da22ac2b8b06fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95f14cded5744864a8da22ac2b8b06fa" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 42355 } }
I20240501 07:58:33.113036 2685 leader_election.cc:302] T 00000000000000000000000000000000 P 95f14cded5744864a8da22ac2b8b06fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 95f14cded5744864a8da22ac2b8b06fa; no voters:
I20240501 07:58:33.113329 2685 leader_election.cc:288] T 00000000000000000000000000000000 P 95f14cded5744864a8da22ac2b8b06fa [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 07:58:33.113430 2690 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 95f14cded5744864a8da22ac2b8b06fa [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:33.113622 2690 raft_consensus.cc:686] T 00000000000000000000000000000000 P 95f14cded5744864a8da22ac2b8b06fa [term 1 LEADER]: Becoming Leader. State: Replica: 95f14cded5744864a8da22ac2b8b06fa, State: Running, Role: LEADER
I20240501 07:58:33.113984 2690 consensus_queue.cc:237] T 00000000000000000000000000000000 P 95f14cded5744864a8da22ac2b8b06fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95f14cded5744864a8da22ac2b8b06fa" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 42355 } }
I20240501 07:58:33.114488 2685 sys_catalog.cc:564] T 00000000000000000000000000000000 P 95f14cded5744864a8da22ac2b8b06fa [sys.catalog]: configured and running, proceeding with master startup.
I20240501 07:58:33.115991 2692 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95f14cded5744864a8da22ac2b8b06fa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 95f14cded5744864a8da22ac2b8b06fa. Latest consensus state: current_term: 1 leader_uuid: "95f14cded5744864a8da22ac2b8b06fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95f14cded5744864a8da22ac2b8b06fa" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 42355 } } }
I20240501 07:58:33.116091 2692 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95f14cded5744864a8da22ac2b8b06fa [sys.catalog]: This master's current role is: LEADER
I20240501 07:58:33.116052 2691 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95f14cded5744864a8da22ac2b8b06fa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "95f14cded5744864a8da22ac2b8b06fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95f14cded5744864a8da22ac2b8b06fa" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 42355 } } }
I20240501 07:58:33.116151 2691 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95f14cded5744864a8da22ac2b8b06fa [sys.catalog]: This master's current role is: LEADER
I20240501 07:58:33.116478 2700 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 07:58:33.119496 2700 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 07:58:33.125329 2700 catalog_manager.cc:1347] Generated new cluster ID: acedc10d66424e13a1405430fb36d2ae
I20240501 07:58:33.125438 2700 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 07:58:33.133416 2700 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 07:58:33.134393 2700 catalog_manager.cc:1504] Loading token signing keys...
I20240501 07:58:33.147212 2700 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 95f14cded5744864a8da22ac2b8b06fa: Generated new TSK 0
I20240501 07:58:33.148051 2700 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 07:58:33.158775 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tbgEt/build/debug/bin/kudu
/tmp/dist-test-task7tbgEt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.129:0
--local_ip_for_outbound_sockets=127.0.99.129
--webserver_interface=127.0.99.129
--webserver_port=0
--tserver_master_addrs=127.0.99.190:42355
--builtin_ntp_servers=127.0.99.148:34309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240501 07:58:33.274214 2709 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:58:33.274467 2709 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 07:58:33.274518 2709 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:58:33.277987 2709 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 07:58:33.278079 2709 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 07:58:33.278155 2709 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.129
I20240501 07:58:33.282267 2709 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:34309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.0.99.129
--webserver_port=0
--tserver_master_addrs=127.0.99.190:42355
--never_fsync=true
--heap_profile_path=/tmp/kudu.2709
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.129
--log_dir=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 07:43:16 UTC on 1a15035a6830
build id 69
I20240501 07:58:33.283282 2709 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 07:58:33.284523 2709 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:58:33.293803 2714 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:58:33.294534 2715 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:58:33.295310 2717 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:58:33.296386 2709 server_base.cc:1034] running on GCE node
I20240501 07:58:33.296854 2709 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 07:58:33.297541 2709 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 07:58:33.298722 2709 hybrid_clock.cc:648] HybridClock initialized: now 1714550313298706 us; error 32 us; skew 500 ppm
I20240501 07:58:33.300881 2709 webserver.cc:460] Webserver started at http://127.0.99.129:45175/ using document root <none> and password file <none>
I20240501 07:58:33.301515 2709 fs_manager.cc:359] Metadata directory not provided
I20240501 07:58:33.301580 2709 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:58:33.301772 2709 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 07:58:33.303573 2709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "809d92b226a94aa59287ddc81c7ff77b"
format_stamp: "Formatted at 2024-05-01 07:58:33 on dist-test-slave-40g1"
I20240501 07:58:33.304117 2709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "809d92b226a94aa59287ddc81c7ff77b"
format_stamp: "Formatted at 2024-05-01 07:58:33 on dist-test-slave-40g1"
I20240501 07:58:33.307803 2709 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20240501 07:58:33.310290 2723 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:33.311489 2709 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240501 07:58:33.311619 2709 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "809d92b226a94aa59287ddc81c7ff77b"
format_stamp: "Formatted at 2024-05-01 07:58:33 on dist-test-slave-40g1"
I20240501 07:58:33.311736 2709 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:58:33.331187 2709 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:58:33.331949 2709 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 07:58:33.332127 2709 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:58:33.332707 2709 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 07:58:33.333715 2709 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 07:58:33.333766 2709 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:33.333825 2709 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 07:58:33.333854 2709 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:33.343336 2709 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.129:33451
I20240501 07:58:33.343359 2836 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.129:33451 every 8 connection(s)
I20240501 07:58:33.344398 2709 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240501 07:58:33.345002 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tbgEt/build/debug/bin/kudu as pid 2709
I20240501 07:58:33.345101 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240501 07:58:33.347632 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tbgEt/build/debug/bin/kudu
/tmp/dist-test-task7tbgEt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.130:0
--local_ip_for_outbound_sockets=127.0.99.130
--webserver_interface=127.0.99.130
--webserver_port=0
--tserver_master_addrs=127.0.99.190:42355
--builtin_ntp_servers=127.0.99.148:34309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 07:58:33.354480 2837 heartbeater.cc:347] Connected to a master server at 127.0.99.190:42355
I20240501 07:58:33.354744 2837 heartbeater.cc:464] Registering TS with master...
I20240501 07:58:33.355322 2837 heartbeater.cc:511] Master 127.0.99.190:42355 requested a full tablet report, sending...
I20240501 07:58:33.357064 2649 ts_manager.cc:194] Registered new tserver with Master: 809d92b226a94aa59287ddc81c7ff77b (127.0.99.129:33451)
I20240501 07:58:33.358877 2649 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.129:43703
W20240501 07:58:33.465507 2840 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:58:33.465768 2840 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 07:58:33.465817 2840 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:58:33.469143 2840 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 07:58:33.469229 2840 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 07:58:33.469302 2840 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.130
I20240501 07:58:33.473362 2840 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:34309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.0.99.130
--webserver_port=0
--tserver_master_addrs=127.0.99.190:42355
--never_fsync=true
--heap_profile_path=/tmp/kudu.2840
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.130
--log_dir=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 07:43:16 UTC on 1a15035a6830
build id 69
I20240501 07:58:33.474344 2840 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 07:58:33.475497 2840 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:58:33.484766 2845 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:58:33.485489 2848 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:58:33.486500 2846 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:58:33.487489 2840 server_base.cc:1034] running on GCE node
I20240501 07:58:33.487920 2840 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 07:58:33.488546 2840 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 07:58:33.489699 2840 hybrid_clock.cc:648] HybridClock initialized: now 1714550313489673 us; error 43 us; skew 500 ppm
I20240501 07:58:33.491832 2840 webserver.cc:460] Webserver started at http://127.0.99.130:45177/ using document root <none> and password file <none>
I20240501 07:58:33.492431 2840 fs_manager.cc:359] Metadata directory not provided
I20240501 07:58:33.492491 2840 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:58:33.492697 2840 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 07:58:33.494570 2840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "853826bd6abf488a8895392f3470a546"
format_stamp: "Formatted at 2024-05-01 07:58:33 on dist-test-slave-40g1"
I20240501 07:58:33.495069 2840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "853826bd6abf488a8895392f3470a546"
format_stamp: "Formatted at 2024-05-01 07:58:33 on dist-test-slave-40g1"
I20240501 07:58:33.498687 2840 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.005s
I20240501 07:58:33.501027 2854 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:33.502110 2840 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.002s
I20240501 07:58:33.502223 2840 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "853826bd6abf488a8895392f3470a546"
format_stamp: "Formatted at 2024-05-01 07:58:33 on dist-test-slave-40g1"
I20240501 07:58:33.502315 2840 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:58:33.520931 2840 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:58:33.521704 2840 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 07:58:33.521870 2840 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:58:33.522429 2840 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 07:58:33.523346 2840 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 07:58:33.523401 2840 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:33.523459 2840 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 07:58:33.523483 2840 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:33.532548 2840 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.130:39801
I20240501 07:58:33.532631 2967 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.130:39801 every 8 connection(s)
I20240501 07:58:33.533633 2840 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240501 07:58:33.534035 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tbgEt/build/debug/bin/kudu as pid 2840
I20240501 07:58:33.534145 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240501 07:58:33.536446 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tbgEt/build/debug/bin/kudu
/tmp/dist-test-task7tbgEt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.131:0
--local_ip_for_outbound_sockets=127.0.99.131
--webserver_interface=127.0.99.131
--webserver_port=0
--tserver_master_addrs=127.0.99.190:42355
--builtin_ntp_servers=127.0.99.148:34309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 07:58:33.543684 2968 heartbeater.cc:347] Connected to a master server at 127.0.99.190:42355
I20240501 07:58:33.543967 2968 heartbeater.cc:464] Registering TS with master...
I20240501 07:58:33.544596 2968 heartbeater.cc:511] Master 127.0.99.190:42355 requested a full tablet report, sending...
I20240501 07:58:33.545800 2649 ts_manager.cc:194] Registered new tserver with Master: 853826bd6abf488a8895392f3470a546 (127.0.99.130:39801)
I20240501 07:58:33.546420 2649 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.130:56023
W20240501 07:58:33.654975 2971 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:58:33.655249 2971 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 07:58:33.655323 2971 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:58:33.658602 2971 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 07:58:33.658699 2971 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 07:58:33.658795 2971 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.131
I20240501 07:58:33.662722 2971 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:34309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.0.99.131
--webserver_port=0
--tserver_master_addrs=127.0.99.190:42355
--never_fsync=true
--heap_profile_path=/tmp/kudu.2971
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.131
--log_dir=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 07:43:16 UTC on 1a15035a6830
build id 69
I20240501 07:58:33.663775 2971 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 07:58:33.665026 2971 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:58:33.674381 2979 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:58:33.675104 2977 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:58:33.676203 2976 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:58:33.677176 2971 server_base.cc:1034] running on GCE node
I20240501 07:58:33.677731 2971 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 07:58:33.678414 2971 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 07:58:33.679579 2971 hybrid_clock.cc:648] HybridClock initialized: now 1714550313679573 us; error 44 us; skew 500 ppm
I20240501 07:58:33.681775 2971 webserver.cc:460] Webserver started at http://127.0.99.131:42647/ using document root <none> and password file <none>
I20240501 07:58:33.682391 2971 fs_manager.cc:359] Metadata directory not provided
I20240501 07:58:33.682447 2971 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:58:33.682615 2971 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 07:58:33.684358 2971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "9846630212094f49930ef629ee885368"
format_stamp: "Formatted at 2024-05-01 07:58:33 on dist-test-slave-40g1"
I20240501 07:58:33.684839 2971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "9846630212094f49930ef629ee885368"
format_stamp: "Formatted at 2024-05-01 07:58:33 on dist-test-slave-40g1"
I20240501 07:58:33.688373 2971 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20240501 07:58:33.690881 2985 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:33.692080 2971 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20240501 07:58:33.692196 2971 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "9846630212094f49930ef629ee885368"
format_stamp: "Formatted at 2024-05-01 07:58:33 on dist-test-slave-40g1"
I20240501 07:58:33.692301 2971 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:58:33.705216 2971 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:58:33.705976 2971 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 07:58:33.706140 2971 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:58:33.706713 2971 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 07:58:33.707662 2971 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 07:58:33.707713 2971 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:33.707772 2971 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 07:58:33.707796 2971 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:33.716753 2971 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.131:33467
I20240501 07:58:33.716827 3098 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.131:33467 every 8 connection(s)
I20240501 07:58:33.717813 2971 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240501 07:58:33.722723 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tbgEt/build/debug/bin/kudu as pid 2971
I20240501 07:58:33.722820 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240501 07:58:33.724866 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tbgEt/build/debug/bin/kudu
/tmp/dist-test-task7tbgEt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.132:0
--local_ip_for_outbound_sockets=127.0.99.132
--webserver_interface=127.0.99.132
--webserver_port=0
--tserver_master_addrs=127.0.99.190:42355
--builtin_ntp_servers=127.0.99.148:34309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 07:58:33.727146 3099 heartbeater.cc:347] Connected to a master server at 127.0.99.190:42355
I20240501 07:58:33.727406 3099 heartbeater.cc:464] Registering TS with master...
I20240501 07:58:33.727995 3099 heartbeater.cc:511] Master 127.0.99.190:42355 requested a full tablet report, sending...
I20240501 07:58:33.729205 2649 ts_manager.cc:194] Registered new tserver with Master: 9846630212094f49930ef629ee885368 (127.0.99.131:33467)
I20240501 07:58:33.730069 2649 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.131:50687
W20240501 07:58:33.837515 3102 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:58:33.837770 3102 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 07:58:33.837838 3102 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:58:33.841302 3102 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 07:58:33.841385 3102 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 07:58:33.841456 3102 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.132
I20240501 07:58:33.845280 3102 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:34309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.0.99.132
--webserver_port=0
--tserver_master_addrs=127.0.99.190:42355
--never_fsync=true
--heap_profile_path=/tmp/kudu.3102
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.132
--log_dir=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 07:43:16 UTC on 1a15035a6830
build id 69
I20240501 07:58:33.846349 3102 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 07:58:33.847479 3102 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:58:33.856532 3107 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:58:33.857167 3108 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:58:33.858355 3110 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:58:33.859208 3102 server_base.cc:1034] running on GCE node
I20240501 07:58:33.859642 3102 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 07:58:33.860255 3102 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 07:58:33.861407 3102 hybrid_clock.cc:648] HybridClock initialized: now 1714550313861386 us; error 34 us; skew 500 ppm
I20240501 07:58:33.863458 3102 webserver.cc:460] Webserver started at http://127.0.99.132:40409/ using document root <none> and password file <none>
I20240501 07:58:33.864056 3102 fs_manager.cc:359] Metadata directory not provided
I20240501 07:58:33.864122 3102 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:58:33.864312 3102 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 07:58:33.866113 3102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "cb0a6f265d7e4362aea53c916ceb49ec"
format_stamp: "Formatted at 2024-05-01 07:58:33 on dist-test-slave-40g1"
I20240501 07:58:33.866631 3102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "cb0a6f265d7e4362aea53c916ceb49ec"
format_stamp: "Formatted at 2024-05-01 07:58:33 on dist-test-slave-40g1"
I20240501 07:58:33.870136 3102 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240501 07:58:33.872428 3116 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:33.873486 3102 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240501 07:58:33.873600 3102 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "cb0a6f265d7e4362aea53c916ceb49ec"
format_stamp: "Formatted at 2024-05-01 07:58:33 on dist-test-slave-40g1"
I20240501 07:58:33.873696 3102 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:58:33.888029 3102 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:58:33.888758 3102 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 07:58:33.888926 3102 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:58:33.889508 3102 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 07:58:33.890425 3102 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 07:58:33.890472 3102 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:33.890529 3102 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 07:58:33.890554 3102 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:33.899169 3102 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.132:43375
I20240501 07:58:33.899187 3229 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.132:43375 every 8 connection(s)
I20240501 07:58:33.900203 3102 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240501 07:58:33.901118 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tbgEt/build/debug/bin/kudu as pid 3102
I20240501 07:58:33.901216 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240501 07:58:33.903321 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tbgEt/build/debug/bin/kudu
/tmp/dist-test-task7tbgEt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.133:0
--local_ip_for_outbound_sockets=127.0.99.133
--webserver_interface=127.0.99.133
--webserver_port=0
--tserver_master_addrs=127.0.99.190:42355
--builtin_ntp_servers=127.0.99.148:34309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 07:58:33.909945 3230 heartbeater.cc:347] Connected to a master server at 127.0.99.190:42355
I20240501 07:58:33.910202 3230 heartbeater.cc:464] Registering TS with master...
I20240501 07:58:33.910800 3230 heartbeater.cc:511] Master 127.0.99.190:42355 requested a full tablet report, sending...
I20240501 07:58:33.911826 2649 ts_manager.cc:194] Registered new tserver with Master: cb0a6f265d7e4362aea53c916ceb49ec (127.0.99.132:43375)
I20240501 07:58:33.912381 2649 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.132:49511
W20240501 07:58:34.016920 3233 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:58:34.017215 3233 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 07:58:34.017287 3233 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:58:34.021662 3233 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 07:58:34.021766 3233 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 07:58:34.021867 3233 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.133
I20240501 07:58:34.027209 3233 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:34309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.133:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.0.99.133
--webserver_port=0
--tserver_master_addrs=127.0.99.190:42355
--never_fsync=true
--heap_profile_path=/tmp/kudu.3233
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.133
--log_dir=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 07:43:16 UTC on 1a15035a6830
build id 69
I20240501 07:58:34.028287 3233 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 07:58:34.029487 3233 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:58:34.038339 3238 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:58:34.039014 3241 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:58:34.040093 3239 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:58:34.041047 3233 server_base.cc:1034] running on GCE node
I20240501 07:58:34.041476 3233 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 07:58:34.042089 3233 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 07:58:34.043257 3233 hybrid_clock.cc:648] HybridClock initialized: now 1714550314043235 us; error 38 us; skew 500 ppm
I20240501 07:58:34.045295 3233 webserver.cc:460] Webserver started at http://127.0.99.133:41181/ using document root <none> and password file <none>
I20240501 07:58:34.045869 3233 fs_manager.cc:359] Metadata directory not provided
I20240501 07:58:34.045929 3233 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:58:34.046108 3233 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 07:58:34.047888 3233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "2b98ac09d43149c5bcfd2c0080ea5379"
format_stamp: "Formatted at 2024-05-01 07:58:34 on dist-test-slave-40g1"
I20240501 07:58:34.048357 3233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "2b98ac09d43149c5bcfd2c0080ea5379"
format_stamp: "Formatted at 2024-05-01 07:58:34 on dist-test-slave-40g1"
I20240501 07:58:34.051756 3233 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240501 07:58:34.054003 3247 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:34.055032 3233 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240501 07:58:34.055138 3233 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "2b98ac09d43149c5bcfd2c0080ea5379"
format_stamp: "Formatted at 2024-05-01 07:58:34 on dist-test-slave-40g1"
I20240501 07:58:34.055225 3233 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:58:34.068142 3233 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:58:34.068897 3233 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 07:58:34.069098 3233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:58:34.069672 3233 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 07:58:34.070603 3233 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 07:58:34.070653 3233 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:34.070711 3233 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 07:58:34.070735 3233 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:34.079581 3233 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.133:40367
I20240501 07:58:34.079604 3360 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.133:40367 every 8 connection(s)
I20240501 07:58:34.080561 3233 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 07:58:34.086894 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tbgEt/build/debug/bin/kudu as pid 3233
I20240501 07:58:34.086997 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240501 07:58:34.089228 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tbgEt/build/debug/bin/kudu
/tmp/dist-test-task7tbgEt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.134:0
--local_ip_for_outbound_sockets=127.0.99.134
--webserver_interface=127.0.99.134
--webserver_port=0
--tserver_master_addrs=127.0.99.190:42355
--builtin_ntp_servers=127.0.99.148:34309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 07:58:34.089809 3361 heartbeater.cc:347] Connected to a master server at 127.0.99.190:42355
I20240501 07:58:34.090051 3361 heartbeater.cc:464] Registering TS with master...
I20240501 07:58:34.090596 3361 heartbeater.cc:511] Master 127.0.99.190:42355 requested a full tablet report, sending...
I20240501 07:58:34.091743 2649 ts_manager.cc:194] Registered new tserver with Master: 2b98ac09d43149c5bcfd2c0080ea5379 (127.0.99.133:40367)
I20240501 07:58:34.092307 2649 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.133:45499
W20240501 07:58:34.205085 3364 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:58:34.205358 3364 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 07:58:34.205432 3364 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:58:34.208724 3364 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 07:58:34.208817 3364 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 07:58:34.208921 3364 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.134
I20240501 07:58:34.212759 3364 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:34309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.134:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.0.99.134
--webserver_port=0
--tserver_master_addrs=127.0.99.190:42355
--never_fsync=true
--heap_profile_path=/tmp/kudu.3364
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.134
--log_dir=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 07:43:16 UTC on 1a15035a6830
build id 69
I20240501 07:58:34.213811 3364 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 07:58:34.215016 3364 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:58:34.224244 3370 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:58:34.224999 3372 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:58:34.225983 3369 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:58:34.226969 3364 server_base.cc:1034] running on GCE node
I20240501 07:58:34.227394 3364 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 07:58:34.228034 3364 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 07:58:34.229189 3364 hybrid_clock.cc:648] HybridClock initialized: now 1714550314229165 us; error 40 us; skew 500 ppm
I20240501 07:58:34.231176 3364 webserver.cc:460] Webserver started at http://127.0.99.134:37725/ using document root <none> and password file <none>
I20240501 07:58:34.231762 3364 fs_manager.cc:359] Metadata directory not provided
I20240501 07:58:34.231822 3364 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:58:34.231997 3364 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 07:58:34.233781 3364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "2471afee7ac94728a1506dc907ef8fbb"
format_stamp: "Formatted at 2024-05-01 07:58:34 on dist-test-slave-40g1"
I20240501 07:58:34.234280 3364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "2471afee7ac94728a1506dc907ef8fbb"
format_stamp: "Formatted at 2024-05-01 07:58:34 on dist-test-slave-40g1"
I20240501 07:58:34.238129 3364 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20240501 07:58:34.240679 3378 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:34.241818 3364 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.002s
I20240501 07:58:34.241943 3364 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "2471afee7ac94728a1506dc907ef8fbb"
format_stamp: "Formatted at 2024-05-01 07:58:34 on dist-test-slave-40g1"
I20240501 07:58:34.242043 3364 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:58:34.272125 3364 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:58:34.272814 3364 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 07:58:34.273013 3364 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:58:34.273583 3364 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 07:58:34.274507 3364 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 07:58:34.274560 3364 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:34.274618 3364 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 07:58:34.274643 3364 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:34.283082 3364 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.134:38167
I20240501 07:58:34.283095 3491 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.134:38167 every 8 connection(s)
I20240501 07:58:34.284070 3364 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240501 07:58:34.285661 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tbgEt/build/debug/bin/kudu as pid 3364
I20240501 07:58:34.285776 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240501 07:58:34.288206 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tbgEt/build/debug/bin/kudu
/tmp/dist-test-task7tbgEt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.135:0
--local_ip_for_outbound_sockets=127.0.99.135
--webserver_interface=127.0.99.135
--webserver_port=0
--tserver_master_addrs=127.0.99.190:42355
--builtin_ntp_servers=127.0.99.148:34309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 07:58:34.294155 3492 heartbeater.cc:347] Connected to a master server at 127.0.99.190:42355
I20240501 07:58:34.294466 3492 heartbeater.cc:464] Registering TS with master...
I20240501 07:58:34.295238 3492 heartbeater.cc:511] Master 127.0.99.190:42355 requested a full tablet report, sending...
I20240501 07:58:34.296489 2649 ts_manager.cc:194] Registered new tserver with Master: 2471afee7ac94728a1506dc907ef8fbb (127.0.99.134:38167)
I20240501 07:58:34.297354 2649 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.134:49019
I20240501 07:58:34.361593 2837 heartbeater.cc:503] Master 127.0.99.190:42355 was elected leader, sending a full tablet report...
W20240501 07:58:34.406430 3495 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:58:34.406697 3495 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 07:58:34.406755 3495 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:58:34.410076 3495 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 07:58:34.410161 3495 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 07:58:34.410256 3495 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.135
I20240501 07:58:34.414175 3495 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:34309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.135:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.0.99.135
--webserver_port=0
--tserver_master_addrs=127.0.99.190:42355
--never_fsync=true
--heap_profile_path=/tmp/kudu.3495
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.135
--log_dir=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 07:43:16 UTC on 1a15035a6830
build id 69
I20240501 07:58:34.415221 3495 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 07:58:34.416471 3495 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:58:34.425715 3500 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:58:34.426671 3501 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:58:34.427572 3503 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:58:34.428930 3495 server_base.cc:1034] running on GCE node
I20240501 07:58:34.429533 3495 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 07:58:34.430186 3495 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 07:58:34.431350 3495 hybrid_clock.cc:648] HybridClock initialized: now 1714550314431323 us; error 50 us; skew 500 ppm
I20240501 07:58:34.433485 3495 webserver.cc:460] Webserver started at http://127.0.99.135:42261/ using document root <none> and password file <none>
I20240501 07:58:34.434079 3495 fs_manager.cc:359] Metadata directory not provided
I20240501 07:58:34.434144 3495 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:58:34.434360 3495 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 07:58:34.436147 3495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "98526b95c3e94668a1673fe660495876"
format_stamp: "Formatted at 2024-05-01 07:58:34 on dist-test-slave-40g1"
I20240501 07:58:34.436645 3495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "98526b95c3e94668a1673fe660495876"
format_stamp: "Formatted at 2024-05-01 07:58:34 on dist-test-slave-40g1"
I20240501 07:58:34.440219 3495 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20240501 07:58:34.442649 3509 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:34.443706 3495 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240501 07:58:34.443825 3495 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "98526b95c3e94668a1673fe660495876"
format_stamp: "Formatted at 2024-05-01 07:58:34 on dist-test-slave-40g1"
I20240501 07:58:34.443915 3495 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:58:34.471642 3495 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:58:34.472362 3495 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 07:58:34.472523 3495 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:58:34.473127 3495 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 07:58:34.474051 3495 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 07:58:34.474100 3495 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:34.474160 3495 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 07:58:34.474186 3495 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:34.483580 3495 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.135:39073
I20240501 07:58:34.483589 3622 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.135:39073 every 8 connection(s)
I20240501 07:58:34.484581 3495 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240501 07:58:34.484941 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tbgEt/build/debug/bin/kudu as pid 3495
I20240501 07:58:34.485059 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240501 07:58:34.493445 3623 heartbeater.cc:347] Connected to a master server at 127.0.99.190:42355
I20240501 07:58:34.493690 3623 heartbeater.cc:464] Registering TS with master...
I20240501 07:58:34.494240 3623 heartbeater.cc:511] Master 127.0.99.190:42355 requested a full tablet report, sending...
I20240501 07:58:34.495271 2649 ts_manager.cc:194] Registered new tserver with Master: 98526b95c3e94668a1673fe660495876 (127.0.99.135:39073)
I20240501 07:58:34.495860 2649 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.135:40721
I20240501 07:58:34.499258 398 external_mini_cluster.cc:930] 7 TS(s) registered with all masters
I20240501 07:58:34.508452 2649 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42472:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240501 07:58:34.525185 3295 tablet_service.cc:1467] Processing CreateTablet for tablet c4d49398b37446c383dc0be576a3263f (DEFAULT_TABLE table=TestTable [id=57abfa918c984fb6b31797a22cd74f51]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:34.526185 3295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4d49398b37446c383dc0be576a3263f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:34.531549 3637 tablet_bootstrap.cc:492] T c4d49398b37446c383dc0be576a3263f P 2b98ac09d43149c5bcfd2c0080ea5379: Bootstrap starting.
I20240501 07:58:34.533887 3637 tablet_bootstrap.cc:654] T c4d49398b37446c383dc0be576a3263f P 2b98ac09d43149c5bcfd2c0080ea5379: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:34.534793 3637 log.cc:824] T c4d49398b37446c383dc0be576a3263f P 2b98ac09d43149c5bcfd2c0080ea5379: Log is configured to *not* fsync() on all Append() calls
I20240501 07:58:34.536746 3637 tablet_bootstrap.cc:492] T c4d49398b37446c383dc0be576a3263f P 2b98ac09d43149c5bcfd2c0080ea5379: No bootstrap required, opened a new log
I20240501 07:58:34.536908 3637 ts_tablet_manager.cc:1367] T c4d49398b37446c383dc0be576a3263f P 2b98ac09d43149c5bcfd2c0080ea5379: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240501 07:58:34.540355 3637 raft_consensus.cc:348] T c4d49398b37446c383dc0be576a3263f P 2b98ac09d43149c5bcfd2c0080ea5379 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b98ac09d43149c5bcfd2c0080ea5379" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 40367 } }
I20240501 07:58:34.540547 3637 raft_consensus.cc:374] T c4d49398b37446c383dc0be576a3263f P 2b98ac09d43149c5bcfd2c0080ea5379 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:34.540583 3637 raft_consensus.cc:729] T c4d49398b37446c383dc0be576a3263f P 2b98ac09d43149c5bcfd2c0080ea5379 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b98ac09d43149c5bcfd2c0080ea5379, State: Initialized, Role: FOLLOWER
I20240501 07:58:34.541076 3637 consensus_queue.cc:260] T c4d49398b37446c383dc0be576a3263f P 2b98ac09d43149c5bcfd2c0080ea5379 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b98ac09d43149c5bcfd2c0080ea5379" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 40367 } }
I20240501 07:58:34.541221 3637 raft_consensus.cc:388] T c4d49398b37446c383dc0be576a3263f P 2b98ac09d43149c5bcfd2c0080ea5379 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 07:58:34.541304 3637 raft_consensus.cc:482] T c4d49398b37446c383dc0be576a3263f P 2b98ac09d43149c5bcfd2c0080ea5379 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 07:58:34.541412 3637 raft_consensus.cc:3037] T c4d49398b37446c383dc0be576a3263f P 2b98ac09d43149c5bcfd2c0080ea5379 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:34.542268 3637 raft_consensus.cc:504] T c4d49398b37446c383dc0be576a3263f P 2b98ac09d43149c5bcfd2c0080ea5379 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b98ac09d43149c5bcfd2c0080ea5379" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 40367 } }
I20240501 07:58:34.542579 3637 leader_election.cc:302] T c4d49398b37446c383dc0be576a3263f P 2b98ac09d43149c5bcfd2c0080ea5379 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2b98ac09d43149c5bcfd2c0080ea5379; no voters:
I20240501 07:58:34.542856 3637 leader_election.cc:288] T c4d49398b37446c383dc0be576a3263f P 2b98ac09d43149c5bcfd2c0080ea5379 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 07:58:34.543013 3639 raft_consensus.cc:2781] T c4d49398b37446c383dc0be576a3263f P 2b98ac09d43149c5bcfd2c0080ea5379 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:34.543255 3639 raft_consensus.cc:686] T c4d49398b37446c383dc0be576a3263f P 2b98ac09d43149c5bcfd2c0080ea5379 [term 1 LEADER]: Becoming Leader. State: Replica: 2b98ac09d43149c5bcfd2c0080ea5379, State: Running, Role: LEADER
I20240501 07:58:34.543465 3361 heartbeater.cc:503] Master 127.0.99.190:42355 was elected leader, sending a full tablet report...
I20240501 07:58:34.543571 3639 consensus_queue.cc:237] T c4d49398b37446c383dc0be576a3263f P 2b98ac09d43149c5bcfd2c0080ea5379 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b98ac09d43149c5bcfd2c0080ea5379" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 40367 } }
I20240501 07:58:34.543726 3637 ts_tablet_manager.cc:1398] T c4d49398b37446c383dc0be576a3263f P 2b98ac09d43149c5bcfd2c0080ea5379: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20240501 07:58:34.547551 2649 catalog_manager.cc:5484] T c4d49398b37446c383dc0be576a3263f P 2b98ac09d43149c5bcfd2c0080ea5379 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2b98ac09d43149c5bcfd2c0080ea5379 (127.0.99.133). New cstate: current_term: 1 leader_uuid: "2b98ac09d43149c5bcfd2c0080ea5379" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b98ac09d43149c5bcfd2c0080ea5379" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 40367 } health_report { overall_health: HEALTHY } } }
I20240501 07:58:34.549260 2968 heartbeater.cc:503] Master 127.0.99.190:42355 was elected leader, sending a full tablet report...
I20240501 07:58:34.560245 398 external_mini_cluster.cc:930] 7 TS(s) registered with all masters
I20240501 07:58:34.561684 398 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 809d92b226a94aa59287ddc81c7ff77b to finish bootstrapping
I20240501 07:58:34.567677 398 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 853826bd6abf488a8895392f3470a546 to finish bootstrapping
I20240501 07:58:34.573197 398 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 9846630212094f49930ef629ee885368 to finish bootstrapping
I20240501 07:58:34.578500 398 ts_itest-base.cc:233] Waiting for 0 tablets on tserver cb0a6f265d7e4362aea53c916ceb49ec to finish bootstrapping
I20240501 07:58:34.584757 398 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 2b98ac09d43149c5bcfd2c0080ea5379 to finish bootstrapping
I20240501 07:58:34.589610 398 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 2471afee7ac94728a1506dc907ef8fbb to finish bootstrapping
I20240501 07:58:34.595110 398 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 98526b95c3e94668a1673fe660495876 to finish bootstrapping
I20240501 07:58:34.601055 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tbgEt/build/debug/bin/kudu with pid 2971
I20240501 07:58:34.606772 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tbgEt/build/debug/bin/kudu with pid 3102
I20240501 07:58:34.612224 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tbgEt/build/debug/bin/kudu with pid 3233
I20240501 07:58:34.618090 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tbgEt/build/debug/bin/kudu with pid 3364
I20240501 07:58:34.623579 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tbgEt/build/debug/bin/kudu with pid 3495
I20240501 07:58:38.379913 2649 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42472:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 07:58:38.392081 2770 tablet_service.cc:1467] Processing CreateTablet for tablet fa6cf4447cfe49ecbbfe94accb24aa97 (DEFAULT_TABLE table=rebalance_test_table_0 [id=176edd724cff404e82ccb10fe1aed740]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:38.392063 2902 tablet_service.cc:1467] Processing CreateTablet for tablet c63214bc45b3424492cc114662f6839e (DEFAULT_TABLE table=rebalance_test_table_0 [id=176edd724cff404e82ccb10fe1aed740]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:38.392063 2771 tablet_service.cc:1467] Processing CreateTablet for tablet 14a68b0c3f984b999ddc7856ca076c8f (DEFAULT_TABLE table=rebalance_test_table_0 [id=176edd724cff404e82ccb10fe1aed740]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:38.393126 2771 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14a68b0c3f984b999ddc7856ca076c8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:38.393126 2902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c63214bc45b3424492cc114662f6839e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:38.393522 2770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa6cf4447cfe49ecbbfe94accb24aa97. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:38.399605 3659 tablet_bootstrap.cc:492] T fa6cf4447cfe49ecbbfe94accb24aa97 P 809d92b226a94aa59287ddc81c7ff77b: Bootstrap starting.
I20240501 07:58:38.401280 3660 tablet_bootstrap.cc:492] T c63214bc45b3424492cc114662f6839e P 853826bd6abf488a8895392f3470a546: Bootstrap starting.
I20240501 07:58:38.401610 3659 tablet_bootstrap.cc:654] T fa6cf4447cfe49ecbbfe94accb24aa97 P 809d92b226a94aa59287ddc81c7ff77b: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:38.402320 3659 log.cc:824] T fa6cf4447cfe49ecbbfe94accb24aa97 P 809d92b226a94aa59287ddc81c7ff77b: Log is configured to *not* fsync() on all Append() calls
I20240501 07:58:38.402981 3660 tablet_bootstrap.cc:654] T c63214bc45b3424492cc114662f6839e P 853826bd6abf488a8895392f3470a546: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:38.403578 3660 log.cc:824] T c63214bc45b3424492cc114662f6839e P 853826bd6abf488a8895392f3470a546: Log is configured to *not* fsync() on all Append() calls
I20240501 07:58:38.403915 3659 tablet_bootstrap.cc:492] T fa6cf4447cfe49ecbbfe94accb24aa97 P 809d92b226a94aa59287ddc81c7ff77b: No bootstrap required, opened a new log
I20240501 07:58:38.404095 3659 ts_tablet_manager.cc:1367] T fa6cf4447cfe49ecbbfe94accb24aa97 P 809d92b226a94aa59287ddc81c7ff77b: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240501 07:58:38.405484 3660 tablet_bootstrap.cc:492] T c63214bc45b3424492cc114662f6839e P 853826bd6abf488a8895392f3470a546: No bootstrap required, opened a new log
I20240501 07:58:38.405699 3660 ts_tablet_manager.cc:1367] T c63214bc45b3424492cc114662f6839e P 853826bd6abf488a8895392f3470a546: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240501 07:58:38.408082 3659 raft_consensus.cc:348] T fa6cf4447cfe49ecbbfe94accb24aa97 P 809d92b226a94aa59287ddc81c7ff77b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "809d92b226a94aa59287ddc81c7ff77b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33451 } }
I20240501 07:58:38.408370 3659 raft_consensus.cc:374] T fa6cf4447cfe49ecbbfe94accb24aa97 P 809d92b226a94aa59287ddc81c7ff77b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:38.408457 3659 raft_consensus.cc:729] T fa6cf4447cfe49ecbbfe94accb24aa97 P 809d92b226a94aa59287ddc81c7ff77b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 809d92b226a94aa59287ddc81c7ff77b, State: Initialized, Role: FOLLOWER
I20240501 07:58:38.409000 3659 consensus_queue.cc:260] T fa6cf4447cfe49ecbbfe94accb24aa97 P 809d92b226a94aa59287ddc81c7ff77b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "809d92b226a94aa59287ddc81c7ff77b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33451 } }
I20240501 07:58:38.409155 3659 raft_consensus.cc:388] T fa6cf4447cfe49ecbbfe94accb24aa97 P 809d92b226a94aa59287ddc81c7ff77b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 07:58:38.409229 3659 raft_consensus.cc:482] T fa6cf4447cfe49ecbbfe94accb24aa97 P 809d92b226a94aa59287ddc81c7ff77b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 07:58:38.409334 3659 raft_consensus.cc:3037] T fa6cf4447cfe49ecbbfe94accb24aa97 P 809d92b226a94aa59287ddc81c7ff77b [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:38.409462 3660 raft_consensus.cc:348] T c63214bc45b3424492cc114662f6839e P 853826bd6abf488a8895392f3470a546 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "853826bd6abf488a8895392f3470a546" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39801 } }
I20240501 07:58:38.409646 3660 raft_consensus.cc:374] T c63214bc45b3424492cc114662f6839e P 853826bd6abf488a8895392f3470a546 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:38.409696 3660 raft_consensus.cc:729] T c63214bc45b3424492cc114662f6839e P 853826bd6abf488a8895392f3470a546 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 853826bd6abf488a8895392f3470a546, State: Initialized, Role: FOLLOWER
I20240501 07:58:38.410109 3660 consensus_queue.cc:260] T c63214bc45b3424492cc114662f6839e P 853826bd6abf488a8895392f3470a546 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "853826bd6abf488a8895392f3470a546" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39801 } }
I20240501 07:58:38.410257 3660 raft_consensus.cc:388] T c63214bc45b3424492cc114662f6839e P 853826bd6abf488a8895392f3470a546 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 07:58:38.410334 3660 raft_consensus.cc:482] T c63214bc45b3424492cc114662f6839e P 853826bd6abf488a8895392f3470a546 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 07:58:38.410442 3660 raft_consensus.cc:3037] T c63214bc45b3424492cc114662f6839e P 853826bd6abf488a8895392f3470a546 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:38.410480 3659 raft_consensus.cc:504] T fa6cf4447cfe49ecbbfe94accb24aa97 P 809d92b226a94aa59287ddc81c7ff77b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "809d92b226a94aa59287ddc81c7ff77b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33451 } }
I20240501 07:58:38.410849 3659 leader_election.cc:302] T fa6cf4447cfe49ecbbfe94accb24aa97 P 809d92b226a94aa59287ddc81c7ff77b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 809d92b226a94aa59287ddc81c7ff77b; no voters:
I20240501 07:58:38.411212 3659 leader_election.cc:288] T fa6cf4447cfe49ecbbfe94accb24aa97 P 809d92b226a94aa59287ddc81c7ff77b [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 07:58:38.411311 3663 raft_consensus.cc:2781] T fa6cf4447cfe49ecbbfe94accb24aa97 P 809d92b226a94aa59287ddc81c7ff77b [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:38.411520 3663 raft_consensus.cc:686] T fa6cf4447cfe49ecbbfe94accb24aa97 P 809d92b226a94aa59287ddc81c7ff77b [term 1 LEADER]: Becoming Leader. State: Replica: 809d92b226a94aa59287ddc81c7ff77b, State: Running, Role: LEADER
I20240501 07:58:38.411672 3660 raft_consensus.cc:504] T c63214bc45b3424492cc114662f6839e P 853826bd6abf488a8895392f3470a546 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "853826bd6abf488a8895392f3470a546" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39801 } }
I20240501 07:58:38.411829 3663 consensus_queue.cc:237] T fa6cf4447cfe49ecbbfe94accb24aa97 P 809d92b226a94aa59287ddc81c7ff77b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "809d92b226a94aa59287ddc81c7ff77b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33451 } }
I20240501 07:58:38.412209 3659 ts_tablet_manager.cc:1398] T fa6cf4447cfe49ecbbfe94accb24aa97 P 809d92b226a94aa59287ddc81c7ff77b: Time spent starting tablet: real 0.008s user 0.009s sys 0.000s
I20240501 07:58:38.412367 3660 leader_election.cc:302] T c63214bc45b3424492cc114662f6839e P 853826bd6abf488a8895392f3470a546 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 853826bd6abf488a8895392f3470a546; no voters:
I20240501 07:58:38.412460 3659 tablet_bootstrap.cc:492] T 14a68b0c3f984b999ddc7856ca076c8f P 809d92b226a94aa59287ddc81c7ff77b: Bootstrap starting.
I20240501 07:58:38.412668 3660 leader_election.cc:288] T c63214bc45b3424492cc114662f6839e P 853826bd6abf488a8895392f3470a546 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 07:58:38.412813 3666 raft_consensus.cc:2781] T c63214bc45b3424492cc114662f6839e P 853826bd6abf488a8895392f3470a546 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:38.413066 3666 raft_consensus.cc:686] T c63214bc45b3424492cc114662f6839e P 853826bd6abf488a8895392f3470a546 [term 1 LEADER]: Becoming Leader. State: Replica: 853826bd6abf488a8895392f3470a546, State: Running, Role: LEADER
I20240501 07:58:38.413400 3666 consensus_queue.cc:237] T c63214bc45b3424492cc114662f6839e P 853826bd6abf488a8895392f3470a546 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "853826bd6abf488a8895392f3470a546" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39801 } }
I20240501 07:58:38.413551 3659 tablet_bootstrap.cc:654] T 14a68b0c3f984b999ddc7856ca076c8f P 809d92b226a94aa59287ddc81c7ff77b: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:38.414558 3660 ts_tablet_manager.cc:1398] T c63214bc45b3424492cc114662f6839e P 853826bd6abf488a8895392f3470a546: Time spent starting tablet: real 0.009s user 0.008s sys 0.000s
I20240501 07:58:38.414804 2649 catalog_manager.cc:5484] T fa6cf4447cfe49ecbbfe94accb24aa97 P 809d92b226a94aa59287ddc81c7ff77b reported cstate change: term changed from 0 to 1, leader changed from <none> to 809d92b226a94aa59287ddc81c7ff77b (127.0.99.129). New cstate: current_term: 1 leader_uuid: "809d92b226a94aa59287ddc81c7ff77b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "809d92b226a94aa59287ddc81c7ff77b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33451 } health_report { overall_health: HEALTHY } } }
I20240501 07:58:38.415566 3659 tablet_bootstrap.cc:492] T 14a68b0c3f984b999ddc7856ca076c8f P 809d92b226a94aa59287ddc81c7ff77b: No bootstrap required, opened a new log
I20240501 07:58:38.415652 3659 ts_tablet_manager.cc:1367] T 14a68b0c3f984b999ddc7856ca076c8f P 809d92b226a94aa59287ddc81c7ff77b: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240501 07:58:38.416266 3659 raft_consensus.cc:348] T 14a68b0c3f984b999ddc7856ca076c8f P 809d92b226a94aa59287ddc81c7ff77b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "809d92b226a94aa59287ddc81c7ff77b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33451 } }
I20240501 07:58:38.416352 3659 raft_consensus.cc:374] T 14a68b0c3f984b999ddc7856ca076c8f P 809d92b226a94aa59287ddc81c7ff77b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:38.416379 3659 raft_consensus.cc:729] T 14a68b0c3f984b999ddc7856ca076c8f P 809d92b226a94aa59287ddc81c7ff77b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 809d92b226a94aa59287ddc81c7ff77b, State: Initialized, Role: FOLLOWER
I20240501 07:58:38.416496 3659 consensus_queue.cc:260] T 14a68b0c3f984b999ddc7856ca076c8f P 809d92b226a94aa59287ddc81c7ff77b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "809d92b226a94aa59287ddc81c7ff77b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33451 } }
I20240501 07:58:38.416576 3659 raft_consensus.cc:388] T 14a68b0c3f984b999ddc7856ca076c8f P 809d92b226a94aa59287ddc81c7ff77b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 07:58:38.416605 3659 raft_consensus.cc:482] T 14a68b0c3f984b999ddc7856ca076c8f P 809d92b226a94aa59287ddc81c7ff77b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 07:58:38.416642 3659 raft_consensus.cc:3037] T 14a68b0c3f984b999ddc7856ca076c8f P 809d92b226a94aa59287ddc81c7ff77b [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:38.417618 3659 raft_consensus.cc:504] T 14a68b0c3f984b999ddc7856ca076c8f P 809d92b226a94aa59287ddc81c7ff77b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "809d92b226a94aa59287ddc81c7ff77b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33451 } }
I20240501 07:58:38.417747 3659 leader_election.cc:302] T 14a68b0c3f984b999ddc7856ca076c8f P 809d92b226a94aa59287ddc81c7ff77b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 809d92b226a94aa59287ddc81c7ff77b; no voters:
I20240501 07:58:38.417855 3659 leader_election.cc:288] T 14a68b0c3f984b999ddc7856ca076c8f P 809d92b226a94aa59287ddc81c7ff77b [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 07:58:38.417912 3663 raft_consensus.cc:2781] T 14a68b0c3f984b999ddc7856ca076c8f P 809d92b226a94aa59287ddc81c7ff77b [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:38.417855 2647 catalog_manager.cc:5484] T c63214bc45b3424492cc114662f6839e P 853826bd6abf488a8895392f3470a546 reported cstate change: term changed from 0 to 1, leader changed from <none> to 853826bd6abf488a8895392f3470a546 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "853826bd6abf488a8895392f3470a546" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "853826bd6abf488a8895392f3470a546" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39801 } health_report { overall_health: HEALTHY } } }
I20240501 07:58:38.418007 3663 raft_consensus.cc:686] T 14a68b0c3f984b999ddc7856ca076c8f P 809d92b226a94aa59287ddc81c7ff77b [term 1 LEADER]: Becoming Leader. State: Replica: 809d92b226a94aa59287ddc81c7ff77b, State: Running, Role: LEADER
I20240501 07:58:38.418081 3659 ts_tablet_manager.cc:1398] T 14a68b0c3f984b999ddc7856ca076c8f P 809d92b226a94aa59287ddc81c7ff77b: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 07:58:38.418110 3663 consensus_queue.cc:237] T 14a68b0c3f984b999ddc7856ca076c8f P 809d92b226a94aa59287ddc81c7ff77b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "809d92b226a94aa59287ddc81c7ff77b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33451 } }
I20240501 07:58:38.420171 2649 catalog_manager.cc:5484] T 14a68b0c3f984b999ddc7856ca076c8f P 809d92b226a94aa59287ddc81c7ff77b reported cstate change: term changed from 0 to 1, leader changed from <none> to 809d92b226a94aa59287ddc81c7ff77b (127.0.99.129). New cstate: current_term: 1 leader_uuid: "809d92b226a94aa59287ddc81c7ff77b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "809d92b226a94aa59287ddc81c7ff77b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33451 } health_report { overall_health: HEALTHY } } }
W20240501 07:58:38.538677 3675 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:58:38.539050 3675 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:58:38.542357 3675 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
INSERT report
rows total: 2000
time total: 72.548 ms
time per row: 0.036274 ms
I20240501 07:58:38.663681 2642 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42472:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 07:58:38.669261 2902 tablet_service.cc:1467] Processing CreateTablet for tablet 444c036684fe4852ba6aaab77411e3a4 (DEFAULT_TABLE table=rebalance_test_table_1 [id=8bdc688d20ef4b1d825861547a2f7334]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:38.669346 2771 tablet_service.cc:1467] Processing CreateTablet for tablet db9747650ebf4556ac30e4138821695b (DEFAULT_TABLE table=rebalance_test_table_1 [id=8bdc688d20ef4b1d825861547a2f7334]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:38.669575 2902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 444c036684fe4852ba6aaab77411e3a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:38.669637 2771 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db9747650ebf4556ac30e4138821695b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:38.671350 3660 tablet_bootstrap.cc:492] T 444c036684fe4852ba6aaab77411e3a4 P 853826bd6abf488a8895392f3470a546: Bootstrap starting.
I20240501 07:58:38.671562 3659 tablet_bootstrap.cc:492] T db9747650ebf4556ac30e4138821695b P 809d92b226a94aa59287ddc81c7ff77b: Bootstrap starting.
I20240501 07:58:38.671566 2901 tablet_service.cc:1467] Processing CreateTablet for tablet 0b2690f2189b43f9a45e1a90605623ad (DEFAULT_TABLE table=rebalance_test_table_1 [id=8bdc688d20ef4b1d825861547a2f7334]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:38.671777 2901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b2690f2189b43f9a45e1a90605623ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:38.672333 3659 tablet_bootstrap.cc:654] T db9747650ebf4556ac30e4138821695b P 809d92b226a94aa59287ddc81c7ff77b: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:38.672333 3660 tablet_bootstrap.cc:654] T 444c036684fe4852ba6aaab77411e3a4 P 853826bd6abf488a8895392f3470a546: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:38.673871 3659 tablet_bootstrap.cc:492] T db9747650ebf4556ac30e4138821695b P 809d92b226a94aa59287ddc81c7ff77b: No bootstrap required, opened a new log
I20240501 07:58:38.673914 3660 tablet_bootstrap.cc:492] T 444c036684fe4852ba6aaab77411e3a4 P 853826bd6abf488a8895392f3470a546: No bootstrap required, opened a new log
I20240501 07:58:38.673954 3659 ts_tablet_manager.cc:1367] T db9747650ebf4556ac30e4138821695b P 809d92b226a94aa59287ddc81c7ff77b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:58:38.673991 3660 ts_tablet_manager.cc:1367] T 444c036684fe4852ba6aaab77411e3a4 P 853826bd6abf488a8895392f3470a546: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 07:58:38.674340 3659 raft_consensus.cc:348] T db9747650ebf4556ac30e4138821695b P 809d92b226a94aa59287ddc81c7ff77b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "809d92b226a94aa59287ddc81c7ff77b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33451 } }
I20240501 07:58:38.674429 3659 raft_consensus.cc:374] T db9747650ebf4556ac30e4138821695b P 809d92b226a94aa59287ddc81c7ff77b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:38.674450 3659 raft_consensus.cc:729] T db9747650ebf4556ac30e4138821695b P 809d92b226a94aa59287ddc81c7ff77b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 809d92b226a94aa59287ddc81c7ff77b, State: Initialized, Role: FOLLOWER
I20240501 07:58:38.674484 3660 raft_consensus.cc:348] T 444c036684fe4852ba6aaab77411e3a4 P 853826bd6abf488a8895392f3470a546 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "853826bd6abf488a8895392f3470a546" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39801 } }
I20240501 07:58:38.674557 3659 consensus_queue.cc:260] T db9747650ebf4556ac30e4138821695b P 809d92b226a94aa59287ddc81c7ff77b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "809d92b226a94aa59287ddc81c7ff77b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33451 } }
I20240501 07:58:38.674585 3660 raft_consensus.cc:374] T 444c036684fe4852ba6aaab77411e3a4 P 853826bd6abf488a8895392f3470a546 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:38.674624 3660 raft_consensus.cc:729] T 444c036684fe4852ba6aaab77411e3a4 P 853826bd6abf488a8895392f3470a546 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 853826bd6abf488a8895392f3470a546, State: Initialized, Role: FOLLOWER
I20240501 07:58:38.674634 3659 raft_consensus.cc:388] T db9747650ebf4556ac30e4138821695b P 809d92b226a94aa59287ddc81c7ff77b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 07:58:38.674669 3659 raft_consensus.cc:482] T db9747650ebf4556ac30e4138821695b P 809d92b226a94aa59287ddc81c7ff77b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 07:58:38.674721 3659 raft_consensus.cc:3037] T db9747650ebf4556ac30e4138821695b P 809d92b226a94aa59287ddc81c7ff77b [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:38.674755 3660 consensus_queue.cc:260] T 444c036684fe4852ba6aaab77411e3a4 P 853826bd6abf488a8895392f3470a546 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "853826bd6abf488a8895392f3470a546" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39801 } }
I20240501 07:58:38.674837 3660 raft_consensus.cc:388] T 444c036684fe4852ba6aaab77411e3a4 P 853826bd6abf488a8895392f3470a546 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 07:58:38.674880 3660 raft_consensus.cc:482] T 444c036684fe4852ba6aaab77411e3a4 P 853826bd6abf488a8895392f3470a546 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 07:58:38.674926 3660 raft_consensus.cc:3037] T 444c036684fe4852ba6aaab77411e3a4 P 853826bd6abf488a8895392f3470a546 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:38.675436 3659 raft_consensus.cc:504] T db9747650ebf4556ac30e4138821695b P 809d92b226a94aa59287ddc81c7ff77b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "809d92b226a94aa59287ddc81c7ff77b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33451 } }
I20240501 07:58:38.675542 3659 leader_election.cc:302] T db9747650ebf4556ac30e4138821695b P 809d92b226a94aa59287ddc81c7ff77b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 809d92b226a94aa59287ddc81c7ff77b; no voters:
I20240501 07:58:38.675535 3660 raft_consensus.cc:504] T 444c036684fe4852ba6aaab77411e3a4 P 853826bd6abf488a8895392f3470a546 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "853826bd6abf488a8895392f3470a546" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39801 } }
I20240501 07:58:38.675634 3659 leader_election.cc:288] T db9747650ebf4556ac30e4138821695b P 809d92b226a94aa59287ddc81c7ff77b [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 07:58:38.675648 3660 leader_election.cc:302] T 444c036684fe4852ba6aaab77411e3a4 P 853826bd6abf488a8895392f3470a546 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 853826bd6abf488a8895392f3470a546; no voters:
I20240501 07:58:38.675707 3665 raft_consensus.cc:2781] T db9747650ebf4556ac30e4138821695b P 809d92b226a94aa59287ddc81c7ff77b [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:38.675781 3665 raft_consensus.cc:686] T db9747650ebf4556ac30e4138821695b P 809d92b226a94aa59287ddc81c7ff77b [term 1 LEADER]: Becoming Leader. State: Replica: 809d92b226a94aa59287ddc81c7ff77b, State: Running, Role: LEADER
I20240501 07:58:38.675781 3666 raft_consensus.cc:2781] T 444c036684fe4852ba6aaab77411e3a4 P 853826bd6abf488a8895392f3470a546 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:38.675853 3666 raft_consensus.cc:686] T 444c036684fe4852ba6aaab77411e3a4 P 853826bd6abf488a8895392f3470a546 [term 1 LEADER]: Becoming Leader. State: Replica: 853826bd6abf488a8895392f3470a546, State: Running, Role: LEADER
I20240501 07:58:38.675915 3659 ts_tablet_manager.cc:1398] T db9747650ebf4556ac30e4138821695b P 809d92b226a94aa59287ddc81c7ff77b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:58:38.675886 3665 consensus_queue.cc:237] T db9747650ebf4556ac30e4138821695b P 809d92b226a94aa59287ddc81c7ff77b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "809d92b226a94aa59287ddc81c7ff77b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33451 } }
I20240501 07:58:38.675968 3666 consensus_queue.cc:237] T 444c036684fe4852ba6aaab77411e3a4 P 853826bd6abf488a8895392f3470a546 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "853826bd6abf488a8895392f3470a546" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39801 } }
I20240501 07:58:38.676919 3660 leader_election.cc:288] T 444c036684fe4852ba6aaab77411e3a4 P 853826bd6abf488a8895392f3470a546 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 07:58:38.677594 2649 catalog_manager.cc:5484] T db9747650ebf4556ac30e4138821695b P 809d92b226a94aa59287ddc81c7ff77b reported cstate change: term changed from 0 to 1, leader changed from <none> to 809d92b226a94aa59287ddc81c7ff77b (127.0.99.129). New cstate: current_term: 1 leader_uuid: "809d92b226a94aa59287ddc81c7ff77b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "809d92b226a94aa59287ddc81c7ff77b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33451 } health_report { overall_health: HEALTHY } } }
I20240501 07:58:38.677608 2645 catalog_manager.cc:5484] T 444c036684fe4852ba6aaab77411e3a4 P 853826bd6abf488a8895392f3470a546 reported cstate change: term changed from 0 to 1, leader changed from <none> to 853826bd6abf488a8895392f3470a546 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "853826bd6abf488a8895392f3470a546" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "853826bd6abf488a8895392f3470a546" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39801 } health_report { overall_health: HEALTHY } } }
I20240501 07:58:38.678294 3660 ts_tablet_manager.cc:1398] T 444c036684fe4852ba6aaab77411e3a4 P 853826bd6abf488a8895392f3470a546: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 07:58:38.678442 3660 tablet_bootstrap.cc:492] T 0b2690f2189b43f9a45e1a90605623ad P 853826bd6abf488a8895392f3470a546: Bootstrap starting.
I20240501 07:58:38.679389 3660 tablet_bootstrap.cc:654] T 0b2690f2189b43f9a45e1a90605623ad P 853826bd6abf488a8895392f3470a546: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:38.680687 3660 tablet_bootstrap.cc:492] T 0b2690f2189b43f9a45e1a90605623ad P 853826bd6abf488a8895392f3470a546: No bootstrap required, opened a new log
I20240501 07:58:38.680766 3660 ts_tablet_manager.cc:1367] T 0b2690f2189b43f9a45e1a90605623ad P 853826bd6abf488a8895392f3470a546: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 07:58:38.681237 3660 raft_consensus.cc:348] T 0b2690f2189b43f9a45e1a90605623ad P 853826bd6abf488a8895392f3470a546 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "853826bd6abf488a8895392f3470a546" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39801 } }
I20240501 07:58:38.681360 3660 raft_consensus.cc:374] T 0b2690f2189b43f9a45e1a90605623ad P 853826bd6abf488a8895392f3470a546 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:38.681397 3660 raft_consensus.cc:729] T 0b2690f2189b43f9a45e1a90605623ad P 853826bd6abf488a8895392f3470a546 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 853826bd6abf488a8895392f3470a546, State: Initialized, Role: FOLLOWER
I20240501 07:58:38.681519 3660 consensus_queue.cc:260] T 0b2690f2189b43f9a45e1a90605623ad P 853826bd6abf488a8895392f3470a546 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "853826bd6abf488a8895392f3470a546" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39801 } }
I20240501 07:58:38.681607 3660 raft_consensus.cc:388] T 0b2690f2189b43f9a45e1a90605623ad P 853826bd6abf488a8895392f3470a546 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 07:58:38.681648 3660 raft_consensus.cc:482] T 0b2690f2189b43f9a45e1a90605623ad P 853826bd6abf488a8895392f3470a546 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 07:58:38.681695 3660 raft_consensus.cc:3037] T 0b2690f2189b43f9a45e1a90605623ad P 853826bd6abf488a8895392f3470a546 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:38.682497 3660 raft_consensus.cc:504] T 0b2690f2189b43f9a45e1a90605623ad P 853826bd6abf488a8895392f3470a546 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "853826bd6abf488a8895392f3470a546" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39801 } }
I20240501 07:58:38.682621 3660 leader_election.cc:302] T 0b2690f2189b43f9a45e1a90605623ad P 853826bd6abf488a8895392f3470a546 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 853826bd6abf488a8895392f3470a546; no voters:
I20240501 07:58:38.682703 3660 leader_election.cc:288] T 0b2690f2189b43f9a45e1a90605623ad P 853826bd6abf488a8895392f3470a546 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 07:58:38.682955 3660 ts_tablet_manager.cc:1398] T 0b2690f2189b43f9a45e1a90605623ad P 853826bd6abf488a8895392f3470a546: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 07:58:38.683128 3669 raft_consensus.cc:2781] T 0b2690f2189b43f9a45e1a90605623ad P 853826bd6abf488a8895392f3470a546 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:38.683228 3669 raft_consensus.cc:686] T 0b2690f2189b43f9a45e1a90605623ad P 853826bd6abf488a8895392f3470a546 [term 1 LEADER]: Becoming Leader. State: Replica: 853826bd6abf488a8895392f3470a546, State: Running, Role: LEADER
I20240501 07:58:38.683362 3669 consensus_queue.cc:237] T 0b2690f2189b43f9a45e1a90605623ad P 853826bd6abf488a8895392f3470a546 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "853826bd6abf488a8895392f3470a546" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39801 } }
I20240501 07:58:38.684635 2645 catalog_manager.cc:5484] T 0b2690f2189b43f9a45e1a90605623ad P 853826bd6abf488a8895392f3470a546 reported cstate change: term changed from 0 to 1, leader changed from <none> to 853826bd6abf488a8895392f3470a546 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "853826bd6abf488a8895392f3470a546" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "853826bd6abf488a8895392f3470a546" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39801 } health_report { overall_health: HEALTHY } } }
W20240501 07:58:38.796213 3713 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:58:38.796521 3713 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:58:38.799798 3713 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
INSERT report
rows total: 2000
time total: 59.7002 ms
time per row: 0.0298501 ms
I20240501 07:58:38.902465 2645 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42472:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 07:58:38.908249 2770 tablet_service.cc:1467] Processing CreateTablet for tablet f4277f2f67dd492297a8b36edc366bba (DEFAULT_TABLE table=rebalance_test_table_2 [id=82153012ad124c3398cc16ce35fb1b37]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:38.908234 2771 tablet_service.cc:1467] Processing CreateTablet for tablet 18f0be6c9b944ce1b63600f456f272dd (DEFAULT_TABLE table=rebalance_test_table_2 [id=82153012ad124c3398cc16ce35fb1b37]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:38.908543 2901 tablet_service.cc:1467] Processing CreateTablet for tablet 3264671dc2714e98846bc54938cf358c (DEFAULT_TABLE table=rebalance_test_table_2 [id=82153012ad124c3398cc16ce35fb1b37]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:38.908612 2771 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18f0be6c9b944ce1b63600f456f272dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:38.908727 2770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4277f2f67dd492297a8b36edc366bba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:38.908788 2901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3264671dc2714e98846bc54938cf358c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:38.910622 3660 tablet_bootstrap.cc:492] T 3264671dc2714e98846bc54938cf358c P 853826bd6abf488a8895392f3470a546: Bootstrap starting.
I20240501 07:58:38.910871 3659 tablet_bootstrap.cc:492] T 18f0be6c9b944ce1b63600f456f272dd P 809d92b226a94aa59287ddc81c7ff77b: Bootstrap starting.
I20240501 07:58:38.911612 3660 tablet_bootstrap.cc:654] T 3264671dc2714e98846bc54938cf358c P 853826bd6abf488a8895392f3470a546: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:38.911737 3659 tablet_bootstrap.cc:654] T 18f0be6c9b944ce1b63600f456f272dd P 809d92b226a94aa59287ddc81c7ff77b: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:38.912716 3659 tablet_bootstrap.cc:492] T 18f0be6c9b944ce1b63600f456f272dd P 809d92b226a94aa59287ddc81c7ff77b: No bootstrap required, opened a new log
I20240501 07:58:38.912788 3659 ts_tablet_manager.cc:1367] T 18f0be6c9b944ce1b63600f456f272dd P 809d92b226a94aa59287ddc81c7ff77b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:58:38.912823 3660 tablet_bootstrap.cc:492] T 3264671dc2714e98846bc54938cf358c P 853826bd6abf488a8895392f3470a546: No bootstrap required, opened a new log
I20240501 07:58:38.912914 3660 ts_tablet_manager.cc:1367] T 3264671dc2714e98846bc54938cf358c P 853826bd6abf488a8895392f3470a546: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 07:58:38.913247 3659 raft_consensus.cc:348] T 18f0be6c9b944ce1b63600f456f272dd P 809d92b226a94aa59287ddc81c7ff77b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "809d92b226a94aa59287ddc81c7ff77b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33451 } }
I20240501 07:58:38.913339 3659 raft_consensus.cc:374] T 18f0be6c9b944ce1b63600f456f272dd P 809d92b226a94aa59287ddc81c7ff77b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:38.913292 3660 raft_consensus.cc:348] T 3264671dc2714e98846bc54938cf358c P 853826bd6abf488a8895392f3470a546 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "853826bd6abf488a8895392f3470a546" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39801 } }
I20240501 07:58:38.913362 3659 raft_consensus.cc:729] T 18f0be6c9b944ce1b63600f456f272dd P 809d92b226a94aa59287ddc81c7ff77b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 809d92b226a94aa59287ddc81c7ff77b, State: Initialized, Role: FOLLOWER
I20240501 07:58:38.913378 3660 raft_consensus.cc:374] T 3264671dc2714e98846bc54938cf358c P 853826bd6abf488a8895392f3470a546 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:38.913422 3660 raft_consensus.cc:729] T 3264671dc2714e98846bc54938cf358c P 853826bd6abf488a8895392f3470a546 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 853826bd6abf488a8895392f3470a546, State: Initialized, Role: FOLLOWER
I20240501 07:58:38.913472 3659 consensus_queue.cc:260] T 18f0be6c9b944ce1b63600f456f272dd P 809d92b226a94aa59287ddc81c7ff77b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "809d92b226a94aa59287ddc81c7ff77b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33451 } }
I20240501 07:58:38.913554 3659 raft_consensus.cc:388] T 18f0be6c9b944ce1b63600f456f272dd P 809d92b226a94aa59287ddc81c7ff77b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 07:58:38.913552 3660 consensus_queue.cc:260] T 3264671dc2714e98846bc54938cf358c P 853826bd6abf488a8895392f3470a546 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "853826bd6abf488a8895392f3470a546" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39801 } }
I20240501 07:58:38.913602 3659 raft_consensus.cc:482] T 18f0be6c9b944ce1b63600f456f272dd P 809d92b226a94aa59287ddc81c7ff77b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 07:58:38.913622 3660 raft_consensus.cc:388] T 3264671dc2714e98846bc54938cf358c P 853826bd6abf488a8895392f3470a546 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 07:58:38.913645 3659 raft_consensus.cc:3037] T 18f0be6c9b944ce1b63600f456f272dd P 809d92b226a94aa59287ddc81c7ff77b [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:38.913669 3660 raft_consensus.cc:482] T 3264671dc2714e98846bc54938cf358c P 853826bd6abf488a8895392f3470a546 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 07:58:38.913722 3660 raft_consensus.cc:3037] T 3264671dc2714e98846bc54938cf358c P 853826bd6abf488a8895392f3470a546 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:38.914357 3660 raft_consensus.cc:504] T 3264671dc2714e98846bc54938cf358c P 853826bd6abf488a8895392f3470a546 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "853826bd6abf488a8895392f3470a546" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39801 } }
I20240501 07:58:38.914377 3659 raft_consensus.cc:504] T 18f0be6c9b944ce1b63600f456f272dd P 809d92b226a94aa59287ddc81c7ff77b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "809d92b226a94aa59287ddc81c7ff77b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33451 } }
I20240501 07:58:38.914494 3660 leader_election.cc:302] T 3264671dc2714e98846bc54938cf358c P 853826bd6abf488a8895392f3470a546 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 853826bd6abf488a8895392f3470a546; no voters:
I20240501 07:58:38.914498 3659 leader_election.cc:302] T 18f0be6c9b944ce1b63600f456f272dd P 809d92b226a94aa59287ddc81c7ff77b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 809d92b226a94aa59287ddc81c7ff77b; no voters:
I20240501 07:58:38.914590 3659 leader_election.cc:288] T 18f0be6c9b944ce1b63600f456f272dd P 809d92b226a94aa59287ddc81c7ff77b [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 07:58:38.914664 3664 raft_consensus.cc:2781] T 18f0be6c9b944ce1b63600f456f272dd P 809d92b226a94aa59287ddc81c7ff77b [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:38.914759 3664 raft_consensus.cc:686] T 18f0be6c9b944ce1b63600f456f272dd P 809d92b226a94aa59287ddc81c7ff77b [term 1 LEADER]: Becoming Leader. State: Replica: 809d92b226a94aa59287ddc81c7ff77b, State: Running, Role: LEADER
I20240501 07:58:38.914850 3659 ts_tablet_manager.cc:1398] T 18f0be6c9b944ce1b63600f456f272dd P 809d92b226a94aa59287ddc81c7ff77b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 07:58:38.914871 3664 consensus_queue.cc:237] T 18f0be6c9b944ce1b63600f456f272dd P 809d92b226a94aa59287ddc81c7ff77b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "809d92b226a94aa59287ddc81c7ff77b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33451 } }
I20240501 07:58:38.915002 3659 tablet_bootstrap.cc:492] T f4277f2f67dd492297a8b36edc366bba P 809d92b226a94aa59287ddc81c7ff77b: Bootstrap starting.
I20240501 07:58:38.915869 3659 tablet_bootstrap.cc:654] T f4277f2f67dd492297a8b36edc366bba P 809d92b226a94aa59287ddc81c7ff77b: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:38.915880 3660 leader_election.cc:288] T 3264671dc2714e98846bc54938cf358c P 853826bd6abf488a8895392f3470a546 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 07:58:38.915918 3666 raft_consensus.cc:2781] T 3264671dc2714e98846bc54938cf358c P 853826bd6abf488a8895392f3470a546 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:38.916047 3666 raft_consensus.cc:686] T 3264671dc2714e98846bc54938cf358c P 853826bd6abf488a8895392f3470a546 [term 1 LEADER]: Becoming Leader. State: Replica: 853826bd6abf488a8895392f3470a546, State: Running, Role: LEADER
I20240501 07:58:38.916173 3666 consensus_queue.cc:237] T 3264671dc2714e98846bc54938cf358c P 853826bd6abf488a8895392f3470a546 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "853826bd6abf488a8895392f3470a546" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39801 } }
I20240501 07:58:38.916240 3660 ts_tablet_manager.cc:1398] T 3264671dc2714e98846bc54938cf358c P 853826bd6abf488a8895392f3470a546: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 07:58:38.916466 2645 catalog_manager.cc:5484] T 18f0be6c9b944ce1b63600f456f272dd P 809d92b226a94aa59287ddc81c7ff77b reported cstate change: term changed from 0 to 1, leader changed from <none> to 809d92b226a94aa59287ddc81c7ff77b (127.0.99.129). New cstate: current_term: 1 leader_uuid: "809d92b226a94aa59287ddc81c7ff77b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "809d92b226a94aa59287ddc81c7ff77b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33451 } health_report { overall_health: HEALTHY } } }
I20240501 07:58:38.916922 3659 tablet_bootstrap.cc:492] T f4277f2f67dd492297a8b36edc366bba P 809d92b226a94aa59287ddc81c7ff77b: No bootstrap required, opened a new log
I20240501 07:58:38.917016 3659 ts_tablet_manager.cc:1367] T f4277f2f67dd492297a8b36edc366bba P 809d92b226a94aa59287ddc81c7ff77b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 07:58:38.917454 3659 raft_consensus.cc:348] T f4277f2f67dd492297a8b36edc366bba P 809d92b226a94aa59287ddc81c7ff77b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "809d92b226a94aa59287ddc81c7ff77b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33451 } }
I20240501 07:58:38.917544 3659 raft_consensus.cc:374] T f4277f2f67dd492297a8b36edc366bba P 809d92b226a94aa59287ddc81c7ff77b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:38.917578 3659 raft_consensus.cc:729] T f4277f2f67dd492297a8b36edc366bba P 809d92b226a94aa59287ddc81c7ff77b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 809d92b226a94aa59287ddc81c7ff77b, State: Initialized, Role: FOLLOWER
I20240501 07:58:38.917697 3659 consensus_queue.cc:260] T f4277f2f67dd492297a8b36edc366bba P 809d92b226a94aa59287ddc81c7ff77b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "809d92b226a94aa59287ddc81c7ff77b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33451 } }
I20240501 07:58:38.917774 3659 raft_consensus.cc:388] T f4277f2f67dd492297a8b36edc366bba P 809d92b226a94aa59287ddc81c7ff77b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 07:58:38.917805 3659 raft_consensus.cc:482] T f4277f2f67dd492297a8b36edc366bba P 809d92b226a94aa59287ddc81c7ff77b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 07:58:38.917735 2649 catalog_manager.cc:5484] T 3264671dc2714e98846bc54938cf358c P 853826bd6abf488a8895392f3470a546 reported cstate change: term changed from 0 to 1, leader changed from <none> to 853826bd6abf488a8895392f3470a546 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "853826bd6abf488a8895392f3470a546" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "853826bd6abf488a8895392f3470a546" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39801 } health_report { overall_health: HEALTHY } } }
I20240501 07:58:38.917848 3659 raft_consensus.cc:3037] T f4277f2f67dd492297a8b36edc366bba P 809d92b226a94aa59287ddc81c7ff77b [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:38.918757 3659 raft_consensus.cc:504] T f4277f2f67dd492297a8b36edc366bba P 809d92b226a94aa59287ddc81c7ff77b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "809d92b226a94aa59287ddc81c7ff77b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33451 } }
I20240501 07:58:38.918895 3659 leader_election.cc:302] T f4277f2f67dd492297a8b36edc366bba P 809d92b226a94aa59287ddc81c7ff77b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 809d92b226a94aa59287ddc81c7ff77b; no voters:
I20240501 07:58:38.918987 3659 leader_election.cc:288] T f4277f2f67dd492297a8b36edc366bba P 809d92b226a94aa59287ddc81c7ff77b [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 07:58:38.919188 3659 ts_tablet_manager.cc:1398] T f4277f2f67dd492297a8b36edc366bba P 809d92b226a94aa59287ddc81c7ff77b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 07:58:38.919584 3663 raft_consensus.cc:2781] T f4277f2f67dd492297a8b36edc366bba P 809d92b226a94aa59287ddc81c7ff77b [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:38.919668 3663 raft_consensus.cc:686] T f4277f2f67dd492297a8b36edc366bba P 809d92b226a94aa59287ddc81c7ff77b [term 1 LEADER]: Becoming Leader. State: Replica: 809d92b226a94aa59287ddc81c7ff77b, State: Running, Role: LEADER
I20240501 07:58:38.919772 3663 consensus_queue.cc:237] T f4277f2f67dd492297a8b36edc366bba P 809d92b226a94aa59287ddc81c7ff77b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "809d92b226a94aa59287ddc81c7ff77b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33451 } }
I20240501 07:58:38.921581 2642 catalog_manager.cc:5484] T f4277f2f67dd492297a8b36edc366bba P 809d92b226a94aa59287ddc81c7ff77b reported cstate change: term changed from 0 to 1, leader changed from <none> to 809d92b226a94aa59287ddc81c7ff77b (127.0.99.129). New cstate: current_term: 1 leader_uuid: "809d92b226a94aa59287ddc81c7ff77b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "809d92b226a94aa59287ddc81c7ff77b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33451 } health_report { overall_health: HEALTHY } } }
W20240501 07:58:39.033953 3749 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:58:39.034248 3749 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:58:39.037526 3749 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
INSERT report
rows total: 2000
time total: 63.8745 ms
time per row: 0.0319373 ms
I20240501 07:58:39.143921 2645 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42472:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 07:58:39.149693 2902 tablet_service.cc:1467] Processing CreateTablet for tablet 102891a28f95444dac4798b2aace3857 (DEFAULT_TABLE table=rebalance_test_table_3 [id=30a898036c104f329727c3b9bc8b9100]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:39.149648 2770 tablet_service.cc:1467] Processing CreateTablet for tablet 149f1f36b61a4d148c2003caaa7fce0c (DEFAULT_TABLE table=rebalance_test_table_3 [id=30a898036c104f329727c3b9bc8b9100]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:39.149847 2901 tablet_service.cc:1467] Processing CreateTablet for tablet 3e1613b40cab4477952e3c04f2b16e5d (DEFAULT_TABLE table=rebalance_test_table_3 [id=30a898036c104f329727c3b9bc8b9100]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:39.150032 2902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 102891a28f95444dac4798b2aace3857. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:39.150061 2770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 149f1f36b61a4d148c2003caaa7fce0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:39.150228 2901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e1613b40cab4477952e3c04f2b16e5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:39.151839 3660 tablet_bootstrap.cc:492] T 102891a28f95444dac4798b2aace3857 P 853826bd6abf488a8895392f3470a546: Bootstrap starting.
I20240501 07:58:39.152115 3659 tablet_bootstrap.cc:492] T 149f1f36b61a4d148c2003caaa7fce0c P 809d92b226a94aa59287ddc81c7ff77b: Bootstrap starting.
I20240501 07:58:39.152736 3660 tablet_bootstrap.cc:654] T 102891a28f95444dac4798b2aace3857 P 853826bd6abf488a8895392f3470a546: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:39.153066 3659 tablet_bootstrap.cc:654] T 149f1f36b61a4d148c2003caaa7fce0c P 809d92b226a94aa59287ddc81c7ff77b: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:39.153942 3660 tablet_bootstrap.cc:492] T 102891a28f95444dac4798b2aace3857 P 853826bd6abf488a8895392f3470a546: No bootstrap required, opened a new log
I20240501 07:58:39.154012 3660 ts_tablet_manager.cc:1367] T 102891a28f95444dac4798b2aace3857 P 853826bd6abf488a8895392f3470a546: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240501 07:58:39.154145 3659 tablet_bootstrap.cc:492] T 149f1f36b61a4d148c2003caaa7fce0c P 809d92b226a94aa59287ddc81c7ff77b: No bootstrap required, opened a new log
I20240501 07:58:39.154225 3659 ts_tablet_manager.cc:1367] T 149f1f36b61a4d148c2003caaa7fce0c P 809d92b226a94aa59287ddc81c7ff77b: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240501 07:58:39.154404 3660 raft_consensus.cc:348] T 102891a28f95444dac4798b2aace3857 P 853826bd6abf488a8895392f3470a546 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "853826bd6abf488a8895392f3470a546" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39801 } }
I20240501 07:58:39.154493 3660 raft_consensus.cc:374] T 102891a28f95444dac4798b2aace3857 P 853826bd6abf488a8895392f3470a546 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:39.154515 3660 raft_consensus.cc:729] T 102891a28f95444dac4798b2aace3857 P 853826bd6abf488a8895392f3470a546 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 853826bd6abf488a8895392f3470a546, State: Initialized, Role: FOLLOWER
I20240501 07:58:39.154635 3660 consensus_queue.cc:260] T 102891a28f95444dac4798b2aace3857 P 853826bd6abf488a8895392f3470a546 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "853826bd6abf488a8895392f3470a546" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39801 } }
I20240501 07:58:39.154619 3659 raft_consensus.cc:348] T 149f1f36b61a4d148c2003caaa7fce0c P 809d92b226a94aa59287ddc81c7ff77b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "809d92b226a94aa59287ddc81c7ff77b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33451 } }
I20240501 07:58:39.154706 3660 raft_consensus.cc:388] T 102891a28f95444dac4798b2aace3857 P 853826bd6abf488a8895392f3470a546 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 07:58:39.154706 3659 raft_consensus.cc:374] T 149f1f36b61a4d148c2003caaa7fce0c P 809d92b226a94aa59287ddc81c7ff77b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:39.154733 3660 raft_consensus.cc:482] T 102891a28f95444dac4798b2aace3857 P 853826bd6abf488a8895392f3470a546 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 07:58:39.154752 3659 raft_consensus.cc:729] T 149f1f36b61a4d148c2003caaa7fce0c P 809d92b226a94aa59287ddc81c7ff77b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 809d92b226a94aa59287ddc81c7ff77b, State: Initialized, Role: FOLLOWER
I20240501 07:58:39.154781 3660 raft_consensus.cc:3037] T 102891a28f95444dac4798b2aace3857 P 853826bd6abf488a8895392f3470a546 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:39.154888 3659 consensus_queue.cc:260] T 149f1f36b61a4d148c2003caaa7fce0c P 809d92b226a94aa59287ddc81c7ff77b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "809d92b226a94aa59287ddc81c7ff77b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33451 } }
I20240501 07:58:39.154959 3659 raft_consensus.cc:388] T 149f1f36b61a4d148c2003caaa7fce0c P 809d92b226a94aa59287ddc81c7ff77b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 07:58:39.155000 3659 raft_consensus.cc:482] T 149f1f36b61a4d148c2003caaa7fce0c P 809d92b226a94aa59287ddc81c7ff77b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 07:58:39.155048 3659 raft_consensus.cc:3037] T 149f1f36b61a4d148c2003caaa7fce0c P 809d92b226a94aa59287ddc81c7ff77b [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:39.155530 3660 raft_consensus.cc:504] T 102891a28f95444dac4798b2aace3857 P 853826bd6abf488a8895392f3470a546 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "853826bd6abf488a8895392f3470a546" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39801 } }
I20240501 07:58:39.155664 3660 leader_election.cc:302] T 102891a28f95444dac4798b2aace3857 P 853826bd6abf488a8895392f3470a546 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 853826bd6abf488a8895392f3470a546; no voters:
I20240501 07:58:39.155766 3660 leader_election.cc:288] T 102891a28f95444dac4798b2aace3857 P 853826bd6abf488a8895392f3470a546 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 07:58:39.155833 3668 raft_consensus.cc:2781] T 102891a28f95444dac4798b2aace3857 P 853826bd6abf488a8895392f3470a546 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:39.155954 3668 raft_consensus.cc:686] T 102891a28f95444dac4798b2aace3857 P 853826bd6abf488a8895392f3470a546 [term 1 LEADER]: Becoming Leader. State: Replica: 853826bd6abf488a8895392f3470a546, State: Running, Role: LEADER
I20240501 07:58:39.156025 3660 ts_tablet_manager.cc:1398] T 102891a28f95444dac4798b2aace3857 P 853826bd6abf488a8895392f3470a546: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 07:58:39.156064 3668 consensus_queue.cc:237] T 102891a28f95444dac4798b2aace3857 P 853826bd6abf488a8895392f3470a546 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "853826bd6abf488a8895392f3470a546" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39801 } }
I20240501 07:58:39.156140 3660 tablet_bootstrap.cc:492] T 3e1613b40cab4477952e3c04f2b16e5d P 853826bd6abf488a8895392f3470a546: Bootstrap starting.
I20240501 07:58:39.156572 3659 raft_consensus.cc:504] T 149f1f36b61a4d148c2003caaa7fce0c P 809d92b226a94aa59287ddc81c7ff77b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "809d92b226a94aa59287ddc81c7ff77b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33451 } }
I20240501 07:58:39.156694 3659 leader_election.cc:302] T 149f1f36b61a4d148c2003caaa7fce0c P 809d92b226a94aa59287ddc81c7ff77b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 809d92b226a94aa59287ddc81c7ff77b; no voters:
I20240501 07:58:39.157059 3660 tablet_bootstrap.cc:654] T 3e1613b40cab4477952e3c04f2b16e5d P 853826bd6abf488a8895392f3470a546: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:39.157130 3665 raft_consensus.cc:2781] T 149f1f36b61a4d148c2003caaa7fce0c P 809d92b226a94aa59287ddc81c7ff77b [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:39.157224 3665 raft_consensus.cc:686] T 149f1f36b61a4d148c2003caaa7fce0c P 809d92b226a94aa59287ddc81c7ff77b [term 1 LEADER]: Becoming Leader. State: Replica: 809d92b226a94aa59287ddc81c7ff77b, State: Running, Role: LEADER
I20240501 07:58:39.157341 3665 consensus_queue.cc:237] T 149f1f36b61a4d148c2003caaa7fce0c P 809d92b226a94aa59287ddc81c7ff77b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "809d92b226a94aa59287ddc81c7ff77b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33451 } }
I20240501 07:58:39.157575 2645 catalog_manager.cc:5484] T 102891a28f95444dac4798b2aace3857 P 853826bd6abf488a8895392f3470a546 reported cstate change: term changed from 0 to 1, leader changed from <none> to 853826bd6abf488a8895392f3470a546 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "853826bd6abf488a8895392f3470a546" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "853826bd6abf488a8895392f3470a546" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39801 } health_report { overall_health: HEALTHY } } }
I20240501 07:58:39.158232 3660 tablet_bootstrap.cc:492] T 3e1613b40cab4477952e3c04f2b16e5d P 853826bd6abf488a8895392f3470a546: No bootstrap required, opened a new log
I20240501 07:58:39.158311 3660 ts_tablet_manager.cc:1367] T 3e1613b40cab4477952e3c04f2b16e5d P 853826bd6abf488a8895392f3470a546: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240501 07:58:39.158541 3659 leader_election.cc:288] T 149f1f36b61a4d148c2003caaa7fce0c P 809d92b226a94aa59287ddc81c7ff77b [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 07:58:39.158773 3660 raft_consensus.cc:348] T 3e1613b40cab4477952e3c04f2b16e5d P 853826bd6abf488a8895392f3470a546 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "853826bd6abf488a8895392f3470a546" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39801 } }
I20240501 07:58:39.158877 3660 raft_consensus.cc:374] T 3e1613b40cab4477952e3c04f2b16e5d P 853826bd6abf488a8895392f3470a546 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:39.158911 3660 raft_consensus.cc:729] T 3e1613b40cab4477952e3c04f2b16e5d P 853826bd6abf488a8895392f3470a546 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 853826bd6abf488a8895392f3470a546, State: Initialized, Role: FOLLOWER
I20240501 07:58:39.158932 2642 catalog_manager.cc:5484] T 149f1f36b61a4d148c2003caaa7fce0c P 809d92b226a94aa59287ddc81c7ff77b reported cstate change: term changed from 0 to 1, leader changed from <none> to 809d92b226a94aa59287ddc81c7ff77b (127.0.99.129). New cstate: current_term: 1 leader_uuid: "809d92b226a94aa59287ddc81c7ff77b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "809d92b226a94aa59287ddc81c7ff77b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33451 } health_report { overall_health: HEALTHY } } }
I20240501 07:58:39.159020 3660 consensus_queue.cc:260] T 3e1613b40cab4477952e3c04f2b16e5d P 853826bd6abf488a8895392f3470a546 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "853826bd6abf488a8895392f3470a546" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39801 } }
I20240501 07:58:39.159099 3660 raft_consensus.cc:388] T 3e1613b40cab4477952e3c04f2b16e5d P 853826bd6abf488a8895392f3470a546 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 07:58:39.159142 3660 raft_consensus.cc:482] T 3e1613b40cab4477952e3c04f2b16e5d P 853826bd6abf488a8895392f3470a546 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 07:58:39.159179 3660 raft_consensus.cc:3037] T 3e1613b40cab4477952e3c04f2b16e5d P 853826bd6abf488a8895392f3470a546 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:39.159325 3659 ts_tablet_manager.cc:1398] T 149f1f36b61a4d148c2003caaa7fce0c P 809d92b226a94aa59287ddc81c7ff77b: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240501 07:58:39.160040 3660 raft_consensus.cc:504] T 3e1613b40cab4477952e3c04f2b16e5d P 853826bd6abf488a8895392f3470a546 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "853826bd6abf488a8895392f3470a546" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39801 } }
I20240501 07:58:39.160161 3660 leader_election.cc:302] T 3e1613b40cab4477952e3c04f2b16e5d P 853826bd6abf488a8895392f3470a546 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 853826bd6abf488a8895392f3470a546; no voters:
I20240501 07:58:39.160226 3660 leader_election.cc:288] T 3e1613b40cab4477952e3c04f2b16e5d P 853826bd6abf488a8895392f3470a546 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 07:58:39.160290 3669 raft_consensus.cc:2781] T 3e1613b40cab4477952e3c04f2b16e5d P 853826bd6abf488a8895392f3470a546 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:39.160362 3669 raft_consensus.cc:686] T 3e1613b40cab4477952e3c04f2b16e5d P 853826bd6abf488a8895392f3470a546 [term 1 LEADER]: Becoming Leader. State: Replica: 853826bd6abf488a8895392f3470a546, State: Running, Role: LEADER
I20240501 07:58:39.160446 3660 ts_tablet_manager.cc:1398] T 3e1613b40cab4477952e3c04f2b16e5d P 853826bd6abf488a8895392f3470a546: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 07:58:39.160462 3669 consensus_queue.cc:237] T 3e1613b40cab4477952e3c04f2b16e5d P 853826bd6abf488a8895392f3470a546 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "853826bd6abf488a8895392f3470a546" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39801 } }
I20240501 07:58:39.162695 2645 catalog_manager.cc:5484] T 3e1613b40cab4477952e3c04f2b16e5d P 853826bd6abf488a8895392f3470a546 reported cstate change: term changed from 0 to 1, leader changed from <none> to 853826bd6abf488a8895392f3470a546 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "853826bd6abf488a8895392f3470a546" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "853826bd6abf488a8895392f3470a546" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39801 } health_report { overall_health: HEALTHY } } }
W20240501 07:58:39.276126 3781 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:58:39.276376 3781 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:58:39.279502 3781 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_3'
INSERT report
rows total: 2000
time total: 68.5353 ms
time per row: 0.0342676 ms
I20240501 07:58:39.389216 2642 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42472:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 07:58:39.394790 2901 tablet_service.cc:1467] Processing CreateTablet for tablet ef28ebd14a684f1f9f5d6718bf8284b4 (DEFAULT_TABLE table=rebalance_test_table_4 [id=c04f011aac574a36aceb005abe2af590]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:39.394778 2770 tablet_service.cc:1467] Processing CreateTablet for tablet c64dae3606e049e2a72b00148fffc748 (DEFAULT_TABLE table=rebalance_test_table_4 [id=c04f011aac574a36aceb005abe2af590]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:39.394925 2771 tablet_service.cc:1467] Processing CreateTablet for tablet 243736c7fc354ccd86f3ac42401c61c0 (DEFAULT_TABLE table=rebalance_test_table_4 [id=c04f011aac574a36aceb005abe2af590]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:39.395110 2770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c64dae3606e049e2a72b00148fffc748. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:39.395112 2901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef28ebd14a684f1f9f5d6718bf8284b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:39.395406 2771 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 243736c7fc354ccd86f3ac42401c61c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:39.397212 3659 tablet_bootstrap.cc:492] T 243736c7fc354ccd86f3ac42401c61c0 P 809d92b226a94aa59287ddc81c7ff77b: Bootstrap starting.
I20240501 07:58:39.397504 3660 tablet_bootstrap.cc:492] T ef28ebd14a684f1f9f5d6718bf8284b4 P 853826bd6abf488a8895392f3470a546: Bootstrap starting.
I20240501 07:58:39.398104 3659 tablet_bootstrap.cc:654] T 243736c7fc354ccd86f3ac42401c61c0 P 809d92b226a94aa59287ddc81c7ff77b: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:39.398272 3660 tablet_bootstrap.cc:654] T ef28ebd14a684f1f9f5d6718bf8284b4 P 853826bd6abf488a8895392f3470a546: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:39.399225 3659 tablet_bootstrap.cc:492] T 243736c7fc354ccd86f3ac42401c61c0 P 809d92b226a94aa59287ddc81c7ff77b: No bootstrap required, opened a new log
I20240501 07:58:39.399266 3660 tablet_bootstrap.cc:492] T ef28ebd14a684f1f9f5d6718bf8284b4 P 853826bd6abf488a8895392f3470a546: No bootstrap required, opened a new log
I20240501 07:58:39.399295 3659 ts_tablet_manager.cc:1367] T 243736c7fc354ccd86f3ac42401c61c0 P 809d92b226a94aa59287ddc81c7ff77b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:58:39.399348 3660 ts_tablet_manager.cc:1367] T ef28ebd14a684f1f9f5d6718bf8284b4 P 853826bd6abf488a8895392f3470a546: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240501 07:58:39.399665 3659 raft_consensus.cc:348] T 243736c7fc354ccd86f3ac42401c61c0 P 809d92b226a94aa59287ddc81c7ff77b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "809d92b226a94aa59287ddc81c7ff77b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33451 } }
I20240501 07:58:39.399757 3659 raft_consensus.cc:374] T 243736c7fc354ccd86f3ac42401c61c0 P 809d92b226a94aa59287ddc81c7ff77b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:39.399717 3660 raft_consensus.cc:348] T ef28ebd14a684f1f9f5d6718bf8284b4 P 853826bd6abf488a8895392f3470a546 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "853826bd6abf488a8895392f3470a546" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39801 } }
I20240501 07:58:39.399780 3659 raft_consensus.cc:729] T 243736c7fc354ccd86f3ac42401c61c0 P 809d92b226a94aa59287ddc81c7ff77b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 809d92b226a94aa59287ddc81c7ff77b, State: Initialized, Role: FOLLOWER
I20240501 07:58:39.399803 3660 raft_consensus.cc:374] T ef28ebd14a684f1f9f5d6718bf8284b4 P 853826bd6abf488a8895392f3470a546 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:39.399842 3660 raft_consensus.cc:729] T ef28ebd14a684f1f9f5d6718bf8284b4 P 853826bd6abf488a8895392f3470a546 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 853826bd6abf488a8895392f3470a546, State: Initialized, Role: FOLLOWER
I20240501 07:58:39.399894 3659 consensus_queue.cc:260] T 243736c7fc354ccd86f3ac42401c61c0 P 809d92b226a94aa59287ddc81c7ff77b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "809d92b226a94aa59287ddc81c7ff77b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33451 } }
I20240501 07:58:39.399984 3659 raft_consensus.cc:388] T 243736c7fc354ccd86f3ac42401c61c0 P 809d92b226a94aa59287ddc81c7ff77b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 07:58:39.399986 3660 consensus_queue.cc:260] T ef28ebd14a684f1f9f5d6718bf8284b4 P 853826bd6abf488a8895392f3470a546 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "853826bd6abf488a8895392f3470a546" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39801 } }
I20240501 07:58:39.400027 3659 raft_consensus.cc:482] T 243736c7fc354ccd86f3ac42401c61c0 P 809d92b226a94aa59287ddc81c7ff77b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 07:58:39.400063 3660 raft_consensus.cc:388] T ef28ebd14a684f1f9f5d6718bf8284b4 P 853826bd6abf488a8895392f3470a546 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 07:58:39.400082 3659 raft_consensus.cc:3037] T 243736c7fc354ccd86f3ac42401c61c0 P 809d92b226a94aa59287ddc81c7ff77b [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:39.400104 3660 raft_consensus.cc:482] T ef28ebd14a684f1f9f5d6718bf8284b4 P 853826bd6abf488a8895392f3470a546 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 07:58:39.400141 3660 raft_consensus.cc:3037] T ef28ebd14a684f1f9f5d6718bf8284b4 P 853826bd6abf488a8895392f3470a546 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:39.400810 3659 raft_consensus.cc:504] T 243736c7fc354ccd86f3ac42401c61c0 P 809d92b226a94aa59287ddc81c7ff77b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "809d92b226a94aa59287ddc81c7ff77b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33451 } }
I20240501 07:58:39.400836 3660 raft_consensus.cc:504] T ef28ebd14a684f1f9f5d6718bf8284b4 P 853826bd6abf488a8895392f3470a546 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "853826bd6abf488a8895392f3470a546" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39801 } }
I20240501 07:58:39.400943 3660 leader_election.cc:302] T ef28ebd14a684f1f9f5d6718bf8284b4 P 853826bd6abf488a8895392f3470a546 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 853826bd6abf488a8895392f3470a546; no voters:
I20240501 07:58:39.400938 3659 leader_election.cc:302] T 243736c7fc354ccd86f3ac42401c61c0 P 809d92b226a94aa59287ddc81c7ff77b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 809d92b226a94aa59287ddc81c7ff77b; no voters:
I20240501 07:58:39.401135 3669 raft_consensus.cc:2781] T ef28ebd14a684f1f9f5d6718bf8284b4 P 853826bd6abf488a8895392f3470a546 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:39.401186 3659 leader_election.cc:288] T 243736c7fc354ccd86f3ac42401c61c0 P 809d92b226a94aa59287ddc81c7ff77b [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 07:58:39.401206 3669 raft_consensus.cc:686] T ef28ebd14a684f1f9f5d6718bf8284b4 P 853826bd6abf488a8895392f3470a546 [term 1 LEADER]: Becoming Leader. State: Replica: 853826bd6abf488a8895392f3470a546, State: Running, Role: LEADER
I20240501 07:58:39.401304 3665 raft_consensus.cc:2781] T 243736c7fc354ccd86f3ac42401c61c0 P 809d92b226a94aa59287ddc81c7ff77b [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:39.401311 3669 consensus_queue.cc:237] T ef28ebd14a684f1f9f5d6718bf8284b4 P 853826bd6abf488a8895392f3470a546 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "853826bd6abf488a8895392f3470a546" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39801 } }
I20240501 07:58:39.401376 3665 raft_consensus.cc:686] T 243736c7fc354ccd86f3ac42401c61c0 P 809d92b226a94aa59287ddc81c7ff77b [term 1 LEADER]: Becoming Leader. State: Replica: 809d92b226a94aa59287ddc81c7ff77b, State: Running, Role: LEADER
I20240501 07:58:39.401453 3659 ts_tablet_manager.cc:1398] T 243736c7fc354ccd86f3ac42401c61c0 P 809d92b226a94aa59287ddc81c7ff77b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 07:58:39.401499 3665 consensus_queue.cc:237] T 243736c7fc354ccd86f3ac42401c61c0 P 809d92b226a94aa59287ddc81c7ff77b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "809d92b226a94aa59287ddc81c7ff77b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33451 } }
I20240501 07:58:39.401623 3659 tablet_bootstrap.cc:492] T c64dae3606e049e2a72b00148fffc748 P 809d92b226a94aa59287ddc81c7ff77b: Bootstrap starting.
I20240501 07:58:39.401786 3660 leader_election.cc:288] T ef28ebd14a684f1f9f5d6718bf8284b4 P 853826bd6abf488a8895392f3470a546 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 07:58:39.402129 3660 ts_tablet_manager.cc:1398] T ef28ebd14a684f1f9f5d6718bf8284b4 P 853826bd6abf488a8895392f3470a546: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240501 07:58:39.402562 3659 tablet_bootstrap.cc:654] T c64dae3606e049e2a72b00148fffc748 P 809d92b226a94aa59287ddc81c7ff77b: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:39.402997 2640 catalog_manager.cc:5484] T ef28ebd14a684f1f9f5d6718bf8284b4 P 853826bd6abf488a8895392f3470a546 reported cstate change: term changed from 0 to 1, leader changed from <none> to 853826bd6abf488a8895392f3470a546 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "853826bd6abf488a8895392f3470a546" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "853826bd6abf488a8895392f3470a546" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39801 } health_report { overall_health: HEALTHY } } }
I20240501 07:58:39.403332 2642 catalog_manager.cc:5484] T 243736c7fc354ccd86f3ac42401c61c0 P 809d92b226a94aa59287ddc81c7ff77b reported cstate change: term changed from 0 to 1, leader changed from <none> to 809d92b226a94aa59287ddc81c7ff77b (127.0.99.129). New cstate: current_term: 1 leader_uuid: "809d92b226a94aa59287ddc81c7ff77b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "809d92b226a94aa59287ddc81c7ff77b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33451 } health_report { overall_health: HEALTHY } } }
I20240501 07:58:39.403704 3659 tablet_bootstrap.cc:492] T c64dae3606e049e2a72b00148fffc748 P 809d92b226a94aa59287ddc81c7ff77b: No bootstrap required, opened a new log
I20240501 07:58:39.403786 3659 ts_tablet_manager.cc:1367] T c64dae3606e049e2a72b00148fffc748 P 809d92b226a94aa59287ddc81c7ff77b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 07:58:39.404248 3659 raft_consensus.cc:348] T c64dae3606e049e2a72b00148fffc748 P 809d92b226a94aa59287ddc81c7ff77b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "809d92b226a94aa59287ddc81c7ff77b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33451 } }
I20240501 07:58:39.404335 3659 raft_consensus.cc:374] T c64dae3606e049e2a72b00148fffc748 P 809d92b226a94aa59287ddc81c7ff77b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:39.404366 3659 raft_consensus.cc:729] T c64dae3606e049e2a72b00148fffc748 P 809d92b226a94aa59287ddc81c7ff77b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 809d92b226a94aa59287ddc81c7ff77b, State: Initialized, Role: FOLLOWER
I20240501 07:58:39.404489 3659 consensus_queue.cc:260] T c64dae3606e049e2a72b00148fffc748 P 809d92b226a94aa59287ddc81c7ff77b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "809d92b226a94aa59287ddc81c7ff77b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33451 } }
I20240501 07:58:39.404570 3659 raft_consensus.cc:388] T c64dae3606e049e2a72b00148fffc748 P 809d92b226a94aa59287ddc81c7ff77b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 07:58:39.404613 3659 raft_consensus.cc:482] T c64dae3606e049e2a72b00148fffc748 P 809d92b226a94aa59287ddc81c7ff77b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 07:58:39.404654 3659 raft_consensus.cc:3037] T c64dae3606e049e2a72b00148fffc748 P 809d92b226a94aa59287ddc81c7ff77b [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:39.405647 3659 raft_consensus.cc:504] T c64dae3606e049e2a72b00148fffc748 P 809d92b226a94aa59287ddc81c7ff77b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "809d92b226a94aa59287ddc81c7ff77b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33451 } }
I20240501 07:58:39.405776 3659 leader_election.cc:302] T c64dae3606e049e2a72b00148fffc748 P 809d92b226a94aa59287ddc81c7ff77b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 809d92b226a94aa59287ddc81c7ff77b; no voters:
I20240501 07:58:39.405930 3664 raft_consensus.cc:2781] T c64dae3606e049e2a72b00148fffc748 P 809d92b226a94aa59287ddc81c7ff77b [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:39.406055 3664 raft_consensus.cc:686] T c64dae3606e049e2a72b00148fffc748 P 809d92b226a94aa59287ddc81c7ff77b [term 1 LEADER]: Becoming Leader. State: Replica: 809d92b226a94aa59287ddc81c7ff77b, State: Running, Role: LEADER
I20240501 07:58:39.406168 3664 consensus_queue.cc:237] T c64dae3606e049e2a72b00148fffc748 P 809d92b226a94aa59287ddc81c7ff77b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "809d92b226a94aa59287ddc81c7ff77b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33451 } }
I20240501 07:58:39.405865 3659 leader_election.cc:288] T c64dae3606e049e2a72b00148fffc748 P 809d92b226a94aa59287ddc81c7ff77b [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 07:58:39.406790 3659 ts_tablet_manager.cc:1398] T c64dae3606e049e2a72b00148fffc748 P 809d92b226a94aa59287ddc81c7ff77b: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 07:58:39.408776 2640 catalog_manager.cc:5484] T c64dae3606e049e2a72b00148fffc748 P 809d92b226a94aa59287ddc81c7ff77b reported cstate change: term changed from 0 to 1, leader changed from <none> to 809d92b226a94aa59287ddc81c7ff77b (127.0.99.129). New cstate: current_term: 1 leader_uuid: "809d92b226a94aa59287ddc81c7ff77b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "809d92b226a94aa59287ddc81c7ff77b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33451 } health_report { overall_health: HEALTHY } } }
W20240501 07:58:39.523494 3814 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:58:39.523780 3814 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:58:39.526901 3814 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
INSERT report
rows total: 2000
time total: 73.0668 ms
time per row: 0.0365334 ms
I20240501 07:58:39.644416 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tbgEt/build/debug/bin/kudu
/tmp/dist-test-task7tbgEt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.131:33467
--local_ip_for_outbound_sockets=127.0.99.131
--tserver_master_addrs=127.0.99.190:42355
--webserver_port=42647
--webserver_interface=127.0.99.131
--builtin_ntp_servers=127.0.99.148:34309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240501 07:58:39.753374 3841 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:58:39.753621 3841 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 07:58:39.753667 3841 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:58:39.756768 3841 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 07:58:39.756853 3841 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 07:58:39.756924 3841 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.131
I20240501 07:58:39.760756 3841 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:34309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.131:33467
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.0.99.131
--webserver_port=42647
--tserver_master_addrs=127.0.99.190:42355
--never_fsync=true
--heap_profile_path=/tmp/kudu.3841
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.131
--log_dir=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 07:43:16 UTC on 1a15035a6830
build id 69
I20240501 07:58:39.761782 3841 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 07:58:39.762991 3841 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:58:39.772234 3847 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:58:39.772786 3849 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:58:39.773758 3841 server_base.cc:1034] running on GCE node
W20240501 07:58:39.774705 3846 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:58:39.775131 3841 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 07:58:39.775734 3841 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 07:58:39.776890 3841 hybrid_clock.cc:648] HybridClock initialized: now 1714550319776879 us; error 35 us; skew 500 ppm
I20240501 07:58:39.778878 3841 webserver.cc:460] Webserver started at http://127.0.99.131:42647/ using document root <none> and password file <none>
I20240501 07:58:39.779485 3841 fs_manager.cc:359] Metadata directory not provided
I20240501 07:58:39.779564 3841 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:58:39.782943 3841 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20240501 07:58:39.784741 3855 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:39.785758 3841 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 07:58:39.785871 3841 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "9846630212094f49930ef629ee885368"
format_stamp: "Formatted at 2024-05-01 07:58:33 on dist-test-slave-40g1"
I20240501 07:58:39.786319 3841 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:58:39.801187 3841 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:58:39.801977 3841 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 07:58:39.802160 3841 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:58:39.802712 3841 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 07:58:39.803601 3841 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 07:58:39.803649 3841 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:39.803709 3841 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 07:58:39.803735 3841 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:39.812705 3841 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.131:33467
I20240501 07:58:39.812719 3968 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.131:33467 every 8 connection(s)
I20240501 07:58:39.813804 3841 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240501 07:58:39.820225 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tbgEt/build/debug/bin/kudu as pid 3841
I20240501 07:58:39.820672 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tbgEt/build/debug/bin/kudu
/tmp/dist-test-task7tbgEt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.132:43375
--local_ip_for_outbound_sockets=127.0.99.132
--tserver_master_addrs=127.0.99.190:42355
--webserver_port=40409
--webserver_interface=127.0.99.132
--builtin_ntp_servers=127.0.99.148:34309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 07:58:39.823176 3969 heartbeater.cc:347] Connected to a master server at 127.0.99.190:42355
I20240501 07:58:39.823426 3969 heartbeater.cc:464] Registering TS with master...
I20240501 07:58:39.824115 3969 heartbeater.cc:511] Master 127.0.99.190:42355 requested a full tablet report, sending...
I20240501 07:58:39.825404 2640 ts_manager.cc:194] Re-registered known tserver with Master: 9846630212094f49930ef629ee885368 (127.0.99.131:33467)
I20240501 07:58:39.825987 2640 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.131:41185
W20240501 07:58:39.931277 3972 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:58:39.931527 3972 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 07:58:39.931574 3972 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:58:39.934824 3972 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 07:58:39.934901 3972 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 07:58:39.934999 3972 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.132
I20240501 07:58:39.938833 3972 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:34309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.132:43375
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.0.99.132
--webserver_port=40409
--tserver_master_addrs=127.0.99.190:42355
--never_fsync=true
--heap_profile_path=/tmp/kudu.3972
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.132
--log_dir=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 07:43:16 UTC on 1a15035a6830
build id 69
I20240501 07:58:39.939797 3972 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 07:58:39.940977 3972 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:58:39.949661 3977 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:58:39.950670 3978 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:58:39.951581 3980 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:58:39.952459 3972 server_base.cc:1034] running on GCE node
I20240501 07:58:39.952879 3972 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 07:58:39.953510 3972 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 07:58:39.954663 3972 hybrid_clock.cc:648] HybridClock initialized: now 1714550319954639 us; error 41 us; skew 500 ppm
I20240501 07:58:39.956611 3972 webserver.cc:460] Webserver started at http://127.0.99.132:40409/ using document root <none> and password file <none>
I20240501 07:58:39.957219 3972 fs_manager.cc:359] Metadata directory not provided
I20240501 07:58:39.957305 3972 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:58:39.960678 3972 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20240501 07:58:39.962572 3986 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:39.963725 3972 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20240501 07:58:39.963827 3972 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "cb0a6f265d7e4362aea53c916ceb49ec"
format_stamp: "Formatted at 2024-05-01 07:58:33 on dist-test-slave-40g1"
I20240501 07:58:39.964262 3972 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:58:39.994515 3972 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:58:39.995198 3972 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 07:58:39.995338 3972 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:58:39.995824 3972 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 07:58:39.996677 3972 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 07:58:39.996722 3972 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:39.996762 3972 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 07:58:39.996790 3972 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:40.005342 3972 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.132:43375
I20240501 07:58:40.005363 4099 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.132:43375 every 8 connection(s)
I20240501 07:58:40.006408 3972 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240501 07:58:40.007154 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tbgEt/build/debug/bin/kudu as pid 3972
I20240501 07:58:40.007586 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tbgEt/build/debug/bin/kudu
/tmp/dist-test-task7tbgEt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.133:40367
--local_ip_for_outbound_sockets=127.0.99.133
--tserver_master_addrs=127.0.99.190:42355
--webserver_port=41181
--webserver_interface=127.0.99.133
--builtin_ntp_servers=127.0.99.148:34309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 07:58:40.016988 4100 heartbeater.cc:347] Connected to a master server at 127.0.99.190:42355
I20240501 07:58:40.017410 4100 heartbeater.cc:464] Registering TS with master...
I20240501 07:58:40.018110 4100 heartbeater.cc:511] Master 127.0.99.190:42355 requested a full tablet report, sending...
I20240501 07:58:40.019207 2640 ts_manager.cc:194] Re-registered known tserver with Master: cb0a6f265d7e4362aea53c916ceb49ec (127.0.99.132:43375)
I20240501 07:58:40.019912 2640 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.132:47205
W20240501 07:58:40.125568 4102 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:58:40.125793 4102 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 07:58:40.125838 4102 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:58:40.129025 4102 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 07:58:40.129097 4102 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 07:58:40.129168 4102 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.133
I20240501 07:58:40.132911 4102 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:34309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.133:40367
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.0.99.133
--webserver_port=41181
--tserver_master_addrs=127.0.99.190:42355
--never_fsync=true
--heap_profile_path=/tmp/kudu.4102
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.133
--log_dir=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 07:43:16 UTC on 1a15035a6830
build id 69
I20240501 07:58:40.133881 4102 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 07:58:40.135107 4102 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:58:40.143816 4109 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:58:40.144771 4108 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:58:40.145705 4111 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:58:40.146582 4102 server_base.cc:1034] running on GCE node
I20240501 07:58:40.147003 4102 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 07:58:40.147555 4102 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 07:58:40.148695 4102 hybrid_clock.cc:648] HybridClock initialized: now 1714550320148689 us; error 39 us; skew 500 ppm
I20240501 07:58:40.150480 4102 webserver.cc:460] Webserver started at http://127.0.99.133:41181/ using document root <none> and password file <none>
I20240501 07:58:40.151042 4102 fs_manager.cc:359] Metadata directory not provided
I20240501 07:58:40.151118 4102 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:58:40.154506 4102 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20240501 07:58:40.156405 4117 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:40.157490 4102 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 07:58:40.157593 4102 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "2b98ac09d43149c5bcfd2c0080ea5379"
format_stamp: "Formatted at 2024-05-01 07:58:34 on dist-test-slave-40g1"
I20240501 07:58:40.158028 4102 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:58:40.171535 4102 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:58:40.172192 4102 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 07:58:40.172353 4102 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:58:40.172883 4102 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 07:58:40.174010 4124 ts_tablet_manager.cc:516] Loading tablet metadata (0/1 complete)
I20240501 07:58:40.177258 4102 ts_tablet_manager.cc:559] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20240501 07:58:40.177335 4102 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.003s user 0.000s sys 0.000s
I20240501 07:58:40.177392 4102 ts_tablet_manager.cc:574] Registering tablets (0/1 complete)
I20240501 07:58:40.178943 4102 ts_tablet_manager.cc:590] Registered 1 tablets
I20240501 07:58:40.178992 4102 ts_tablet_manager.cc:569] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20240501 07:58:40.179311 4124 tablet_bootstrap.cc:492] T c4d49398b37446c383dc0be576a3263f P 2b98ac09d43149c5bcfd2c0080ea5379: Bootstrap starting.
I20240501 07:58:40.189244 4102 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.133:40367
I20240501 07:58:40.189306 4231 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.133:40367 every 8 connection(s)
I20240501 07:58:40.190344 4102 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 07:58:40.193601 4124 log.cc:824] T c4d49398b37446c383dc0be576a3263f P 2b98ac09d43149c5bcfd2c0080ea5379: Log is configured to *not* fsync() on all Append() calls
I20240501 07:58:40.195091 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tbgEt/build/debug/bin/kudu as pid 4102
I20240501 07:58:40.195457 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tbgEt/build/debug/bin/kudu
/tmp/dist-test-task7tbgEt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.134:38167
--local_ip_for_outbound_sockets=127.0.99.134
--tserver_master_addrs=127.0.99.190:42355
--webserver_port=37725
--webserver_interface=127.0.99.134
--builtin_ntp_servers=127.0.99.148:34309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 07:58:40.196341 4124 tablet_bootstrap.cc:492] T c4d49398b37446c383dc0be576a3263f P 2b98ac09d43149c5bcfd2c0080ea5379: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:58:40.196686 4124 tablet_bootstrap.cc:492] T c4d49398b37446c383dc0be576a3263f P 2b98ac09d43149c5bcfd2c0080ea5379: Bootstrap complete.
I20240501 07:58:40.198082 4124 ts_tablet_manager.cc:1367] T c4d49398b37446c383dc0be576a3263f P 2b98ac09d43149c5bcfd2c0080ea5379: Time spent bootstrapping tablet: real 0.019s user 0.008s sys 0.010s
I20240501 07:58:40.199846 4232 heartbeater.cc:347] Connected to a master server at 127.0.99.190:42355
I20240501 07:58:40.200088 4232 heartbeater.cc:464] Registering TS with master...
I20240501 07:58:40.200748 4232 heartbeater.cc:511] Master 127.0.99.190:42355 requested a full tablet report, sending...
I20240501 07:58:40.201723 4124 raft_consensus.cc:348] T c4d49398b37446c383dc0be576a3263f P 2b98ac09d43149c5bcfd2c0080ea5379 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b98ac09d43149c5bcfd2c0080ea5379" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 40367 } }
I20240501 07:58:40.202132 4124 raft_consensus.cc:729] T c4d49398b37446c383dc0be576a3263f P 2b98ac09d43149c5bcfd2c0080ea5379 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b98ac09d43149c5bcfd2c0080ea5379, State: Initialized, Role: FOLLOWER
I20240501 07:58:40.202593 2640 ts_manager.cc:194] Re-registered known tserver with Master: 2b98ac09d43149c5bcfd2c0080ea5379 (127.0.99.133:40367)
I20240501 07:58:40.202659 4124 consensus_queue.cc:260] T c4d49398b37446c383dc0be576a3263f P 2b98ac09d43149c5bcfd2c0080ea5379 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b98ac09d43149c5bcfd2c0080ea5379" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 40367 } }
I20240501 07:58:40.202824 4124 raft_consensus.cc:388] T c4d49398b37446c383dc0be576a3263f P 2b98ac09d43149c5bcfd2c0080ea5379 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 07:58:40.202900 4124 raft_consensus.cc:482] T c4d49398b37446c383dc0be576a3263f P 2b98ac09d43149c5bcfd2c0080ea5379 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 07:58:40.203017 4124 raft_consensus.cc:3037] T c4d49398b37446c383dc0be576a3263f P 2b98ac09d43149c5bcfd2c0080ea5379 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:58:40.203454 2640 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.133:33913
I20240501 07:58:40.205265 4124 raft_consensus.cc:504] T c4d49398b37446c383dc0be576a3263f P 2b98ac09d43149c5bcfd2c0080ea5379 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b98ac09d43149c5bcfd2c0080ea5379" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 40367 } }
I20240501 07:58:40.205610 4124 leader_election.cc:302] T c4d49398b37446c383dc0be576a3263f P 2b98ac09d43149c5bcfd2c0080ea5379 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2b98ac09d43149c5bcfd2c0080ea5379; no voters:
I20240501 07:58:40.205971 4124 leader_election.cc:288] T c4d49398b37446c383dc0be576a3263f P 2b98ac09d43149c5bcfd2c0080ea5379 [CANDIDATE]: Term 2 election: Requested vote from peers
I20240501 07:58:40.206048 4237 raft_consensus.cc:2781] T c4d49398b37446c383dc0be576a3263f P 2b98ac09d43149c5bcfd2c0080ea5379 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 07:58:40.206246 4237 raft_consensus.cc:686] T c4d49398b37446c383dc0be576a3263f P 2b98ac09d43149c5bcfd2c0080ea5379 [term 2 LEADER]: Becoming Leader. State: Replica: 2b98ac09d43149c5bcfd2c0080ea5379, State: Running, Role: LEADER
I20240501 07:58:40.206537 4237 consensus_queue.cc:237] T c4d49398b37446c383dc0be576a3263f P 2b98ac09d43149c5bcfd2c0080ea5379 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b98ac09d43149c5bcfd2c0080ea5379" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 40367 } }
I20240501 07:58:40.206678 4232 heartbeater.cc:503] Master 127.0.99.190:42355 was elected leader, sending a full tablet report...
I20240501 07:58:40.207098 4124 ts_tablet_manager.cc:1398] T c4d49398b37446c383dc0be576a3263f P 2b98ac09d43149c5bcfd2c0080ea5379: Time spent starting tablet: real 0.009s user 0.008s sys 0.001s
I20240501 07:58:40.208626 2640 catalog_manager.cc:5484] T c4d49398b37446c383dc0be576a3263f P 2b98ac09d43149c5bcfd2c0080ea5379 reported cstate change: term changed from 1 to 2. New cstate: current_term: 2 leader_uuid: "2b98ac09d43149c5bcfd2c0080ea5379" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b98ac09d43149c5bcfd2c0080ea5379" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 40367 } health_report { overall_health: HEALTHY } } }
W20240501 07:58:40.313548 4236 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:58:40.313853 4236 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 07:58:40.313932 4236 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:58:40.317322 4236 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 07:58:40.317409 4236 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 07:58:40.317505 4236 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.134
I20240501 07:58:40.321326 4236 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:34309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.134:38167
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.0.99.134
--webserver_port=37725
--tserver_master_addrs=127.0.99.190:42355
--never_fsync=true
--heap_profile_path=/tmp/kudu.4236
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.134
--log_dir=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 07:43:16 UTC on 1a15035a6830
build id 69
I20240501 07:58:40.322315 4236 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 07:58:40.323518 4236 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:58:40.332506 4250 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:58:40.333218 4252 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:58:40.334218 4249 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:58:40.335039 4236 server_base.cc:1034] running on GCE node
I20240501 07:58:40.335489 4236 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 07:58:40.336094 4236 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 07:58:40.337241 4236 hybrid_clock.cc:648] HybridClock initialized: now 1714550320337225 us; error 32 us; skew 500 ppm
I20240501 07:58:40.339193 4236 webserver.cc:460] Webserver started at http://127.0.99.134:37725/ using document root <none> and password file <none>
I20240501 07:58:40.339749 4236 fs_manager.cc:359] Metadata directory not provided
I20240501 07:58:40.339831 4236 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:58:40.343225 4236 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20240501 07:58:40.345275 4258 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:40.346341 4236 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 07:58:40.346446 4236 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "2471afee7ac94728a1506dc907ef8fbb"
format_stamp: "Formatted at 2024-05-01 07:58:34 on dist-test-slave-40g1"
I20240501 07:58:40.346932 4236 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:58:40.359725 4236 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:58:40.360406 4236 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 07:58:40.360564 4236 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:58:40.361197 4236 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 07:58:40.362164 4236 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 07:58:40.362218 4236 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:40.362290 4236 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 07:58:40.362316 4236 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:40.373310 4236 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.134:38167
I20240501 07:58:40.373351 4371 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.134:38167 every 8 connection(s)
I20240501 07:58:40.374574 4236 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240501 07:58:40.382695 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tbgEt/build/debug/bin/kudu as pid 4236
I20240501 07:58:40.383119 4372 heartbeater.cc:347] Connected to a master server at 127.0.99.190:42355
I20240501 07:58:40.383149 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tbgEt/build/debug/bin/kudu
/tmp/dist-test-task7tbgEt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.135:39073
--local_ip_for_outbound_sockets=127.0.99.135
--tserver_master_addrs=127.0.99.190:42355
--webserver_port=42261
--webserver_interface=127.0.99.135
--builtin_ntp_servers=127.0.99.148:34309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 07:58:40.383353 4372 heartbeater.cc:464] Registering TS with master...
I20240501 07:58:40.383875 4372 heartbeater.cc:511] Master 127.0.99.190:42355 requested a full tablet report, sending...
I20240501 07:58:40.384917 2640 ts_manager.cc:194] Re-registered known tserver with Master: 2471afee7ac94728a1506dc907ef8fbb (127.0.99.134:38167)
I20240501 07:58:40.385509 2640 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.134:40991
W20240501 07:58:40.496053 4375 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:58:40.496297 4375 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 07:58:40.496343 4375 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:58:40.499521 4375 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 07:58:40.499591 4375 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 07:58:40.499660 4375 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.135
I20240501 07:58:40.503471 4375 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:34309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.135:39073
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.0.99.135
--webserver_port=42261
--tserver_master_addrs=127.0.99.190:42355
--never_fsync=true
--heap_profile_path=/tmp/kudu.4375
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.135
--log_dir=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 07:43:16 UTC on 1a15035a6830
build id 69
I20240501 07:58:40.504456 4375 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 07:58:40.505693 4375 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:58:40.515100 4381 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:58:40.515739 4380 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:58:40.516800 4383 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:58:40.517827 4375 server_base.cc:1034] running on GCE node
I20240501 07:58:40.518254 4375 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 07:58:40.518863 4375 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 07:58:40.520018 4375 hybrid_clock.cc:648] HybridClock initialized: now 1714550320520006 us; error 33 us; skew 500 ppm
I20240501 07:58:40.526374 4375 webserver.cc:460] Webserver started at http://127.0.99.135:42261/ using document root <none> and password file <none>
I20240501 07:58:40.526974 4375 fs_manager.cc:359] Metadata directory not provided
I20240501 07:58:40.527061 4375 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:58:40.530535 4375 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20240501 07:58:40.532482 4389 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:40.533578 4375 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 07:58:40.533689 4375 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "98526b95c3e94668a1673fe660495876"
format_stamp: "Formatted at 2024-05-01 07:58:34 on dist-test-slave-40g1"
I20240501 07:58:40.534126 4375 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:58:40.548177 4375 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:58:40.548875 4375 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 07:58:40.549068 4375 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:58:40.549785 4375 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 07:58:40.550799 4375 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 07:58:40.550846 4375 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:40.550904 4375 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 07:58:40.550930 4375 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:40.560673 4375 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.135:39073
I20240501 07:58:40.560706 4502 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.135:39073 every 8 connection(s)
I20240501 07:58:40.561918 4375 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240501 07:58:40.570271 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tbgEt/build/debug/bin/kudu as pid 4375
I20240501 07:58:40.570784 4503 heartbeater.cc:347] Connected to a master server at 127.0.99.190:42355
I20240501 07:58:40.571043 4503 heartbeater.cc:464] Registering TS with master...
I20240501 07:58:40.571707 4503 heartbeater.cc:511] Master 127.0.99.190:42355 requested a full tablet report, sending...
I20240501 07:58:40.572782 2640 ts_manager.cc:194] Re-registered known tserver with Master: 98526b95c3e94668a1673fe660495876 (127.0.99.135:39073)
I20240501 07:58:40.573364 2640 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.135:54077
I20240501 07:58:40.790377 4166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:58:40.791299 3903 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:40.792062 4306 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:40.794186 2770 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 07:58:40.801801 4437 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:40.804765 2901 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 07:58:40.828543 3969 heartbeater.cc:503] Master 127.0.99.190:42355 was elected leader, sending a full tablet report...
I20240501 07:58:40.833253 4034 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:40.942845 4166 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:58:40.942896 2770 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 07:58:40.943902 4437 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:40.945287 4034 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:40.945601 4306 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:40.946077 2901 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 07:58:40.948897 3903 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:41.002418 2922 consensus_queue.cc:237] T 0b2690f2189b43f9a45e1a90605623ad P 853826bd6abf488a8895392f3470a546 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "853826bd6abf488a8895392f3470a546" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39801 } attrs { replace: true } } peers { permanent_uuid: "cb0a6f265d7e4362aea53c916ceb49ec" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 43375 } attrs { promote: true } }
I20240501 07:58:41.003822 4593 raft_consensus.cc:2932] T 0b2690f2189b43f9a45e1a90605623ad P 853826bd6abf488a8895392f3470a546 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER cb0a6f265d7e4362aea53c916ceb49ec (127.0.99.132) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "853826bd6abf488a8895392f3470a546" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39801 } attrs { replace: true } } peers { permanent_uuid: "cb0a6f265d7e4362aea53c916ceb49ec" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 43375 } attrs { promote: true } } }
I20240501 07:58:41.007138 2644 catalog_manager.cc:5484] T 0b2690f2189b43f9a45e1a90605623ad P 853826bd6abf488a8895392f3470a546 reported cstate change: config changed from index -1 to 4, NON_VOTER cb0a6f265d7e4362aea53c916ceb49ec (127.0.99.132) added. New cstate: current_term: 1 leader_uuid: "853826bd6abf488a8895392f3470a546" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "853826bd6abf488a8895392f3470a546" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39801 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cb0a6f265d7e4362aea53c916ceb49ec" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 43375 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240501 07:58:41.009248 2855 consensus_peers.cc:485] T 0b2690f2189b43f9a45e1a90605623ad P 853826bd6abf488a8895392f3470a546 -> Peer cb0a6f265d7e4362aea53c916ceb49ec (127.0.99.132:43375): Couldn't send request to peer cb0a6f265d7e4362aea53c916ceb49ec. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0b2690f2189b43f9a45e1a90605623ad. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:58:41.015906 2791 consensus_queue.cc:237] T c64dae3606e049e2a72b00148fffc748 P 809d92b226a94aa59287ddc81c7ff77b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "809d92b226a94aa59287ddc81c7ff77b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33451 } attrs { replace: true } } peers { permanent_uuid: "2471afee7ac94728a1506dc907ef8fbb" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 38167 } attrs { promote: true } }
I20240501 07:58:41.017405 4605 raft_consensus.cc:2932] T c64dae3606e049e2a72b00148fffc748 P 809d92b226a94aa59287ddc81c7ff77b [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2471afee7ac94728a1506dc907ef8fbb (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "809d92b226a94aa59287ddc81c7ff77b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33451 } attrs { replace: true } } peers { permanent_uuid: "2471afee7ac94728a1506dc907ef8fbb" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 38167 } attrs { promote: true } } }
I20240501 07:58:41.020215 2644 catalog_manager.cc:5484] T c64dae3606e049e2a72b00148fffc748 P 809d92b226a94aa59287ddc81c7ff77b reported cstate change: config changed from index -1 to 4, NON_VOTER 2471afee7ac94728a1506dc907ef8fbb (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "809d92b226a94aa59287ddc81c7ff77b" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "809d92b226a94aa59287ddc81c7ff77b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33451 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2471afee7ac94728a1506dc907ef8fbb" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 38167 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240501 07:58:41.022650 2727 consensus_peers.cc:485] T c64dae3606e049e2a72b00148fffc748 P 809d92b226a94aa59287ddc81c7ff77b -> Peer 2471afee7ac94728a1506dc907ef8fbb (127.0.99.134:38167): Couldn't send request to peer 2471afee7ac94728a1506dc907ef8fbb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c64dae3606e049e2a72b00148fffc748. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:58:41.023151 4100 heartbeater.cc:503] Master 127.0.99.190:42355 was elected leader, sending a full tablet report...
I20240501 07:58:41.027171 2791 consensus_queue.cc:237] T fa6cf4447cfe49ecbbfe94accb24aa97 P 809d92b226a94aa59287ddc81c7ff77b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "809d92b226a94aa59287ddc81c7ff77b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33451 } attrs { replace: true } } peers { permanent_uuid: "98526b95c3e94668a1673fe660495876" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39073 } attrs { promote: true } }
I20240501 07:58:41.028208 4606 raft_consensus.cc:2932] T fa6cf4447cfe49ecbbfe94accb24aa97 P 809d92b226a94aa59287ddc81c7ff77b [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 98526b95c3e94668a1673fe660495876 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "809d92b226a94aa59287ddc81c7ff77b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33451 } attrs { replace: true } } peers { permanent_uuid: "98526b95c3e94668a1673fe660495876" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39073 } attrs { promote: true } } }
I20240501 07:58:41.031342 2644 catalog_manager.cc:5484] T fa6cf4447cfe49ecbbfe94accb24aa97 P 809d92b226a94aa59287ddc81c7ff77b reported cstate change: config changed from index -1 to 4, NON_VOTER 98526b95c3e94668a1673fe660495876 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "809d92b226a94aa59287ddc81c7ff77b" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "809d92b226a94aa59287ddc81c7ff77b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33451 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "98526b95c3e94668a1673fe660495876" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39073 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240501 07:58:41.033155 2725 consensus_peers.cc:485] T fa6cf4447cfe49ecbbfe94accb24aa97 P 809d92b226a94aa59287ddc81c7ff77b -> Peer 98526b95c3e94668a1673fe660495876 (127.0.99.135:39073): Couldn't send request to peer 98526b95c3e94668a1673fe660495876. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fa6cf4447cfe49ecbbfe94accb24aa97. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:58:41.038461 2922 consensus_queue.cc:237] T 102891a28f95444dac4798b2aace3857 P 853826bd6abf488a8895392f3470a546 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "853826bd6abf488a8895392f3470a546" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39801 } attrs { replace: true } } peers { permanent_uuid: "9846630212094f49930ef629ee885368" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 33467 } attrs { promote: true } }
I20240501 07:58:41.039745 4593 raft_consensus.cc:2932] T 102891a28f95444dac4798b2aace3857 P 853826bd6abf488a8895392f3470a546 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9846630212094f49930ef629ee885368 (127.0.99.131) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "853826bd6abf488a8895392f3470a546" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39801 } attrs { replace: true } } peers { permanent_uuid: "9846630212094f49930ef629ee885368" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 33467 } attrs { promote: true } } }
I20240501 07:58:41.042559 2644 catalog_manager.cc:5484] T 102891a28f95444dac4798b2aace3857 P 853826bd6abf488a8895392f3470a546 reported cstate change: config changed from index -1 to 4, NON_VOTER 9846630212094f49930ef629ee885368 (127.0.99.131) added. New cstate: current_term: 1 leader_uuid: "853826bd6abf488a8895392f3470a546" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "853826bd6abf488a8895392f3470a546" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39801 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9846630212094f49930ef629ee885368" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 33467 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240501 07:58:41.044195 2855 consensus_peers.cc:485] T 102891a28f95444dac4798b2aace3857 P 853826bd6abf488a8895392f3470a546 -> Peer 9846630212094f49930ef629ee885368 (127.0.99.131:33467): Couldn't send request to peer 9846630212094f49930ef629ee885368. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 102891a28f95444dac4798b2aace3857. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:58:41.049746 2791 consensus_queue.cc:237] T 18f0be6c9b944ce1b63600f456f272dd P 809d92b226a94aa59287ddc81c7ff77b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "809d92b226a94aa59287ddc81c7ff77b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33451 } attrs { replace: true } } peers { permanent_uuid: "2b98ac09d43149c5bcfd2c0080ea5379" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 40367 } attrs { promote: true } }
I20240501 07:58:41.050860 4605 raft_consensus.cc:2932] T 18f0be6c9b944ce1b63600f456f272dd P 809d92b226a94aa59287ddc81c7ff77b [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2b98ac09d43149c5bcfd2c0080ea5379 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "809d92b226a94aa59287ddc81c7ff77b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33451 } attrs { replace: true } } peers { permanent_uuid: "2b98ac09d43149c5bcfd2c0080ea5379" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 40367 } attrs { promote: true } } }
I20240501 07:58:41.053850 2649 catalog_manager.cc:5484] T 18f0be6c9b944ce1b63600f456f272dd P 809d92b226a94aa59287ddc81c7ff77b reported cstate change: config changed from index -1 to 4, NON_VOTER 2b98ac09d43149c5bcfd2c0080ea5379 (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "809d92b226a94aa59287ddc81c7ff77b" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "809d92b226a94aa59287ddc81c7ff77b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33451 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2b98ac09d43149c5bcfd2c0080ea5379" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 40367 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240501 07:58:41.055611 2726 consensus_peers.cc:485] T 18f0be6c9b944ce1b63600f456f272dd P 809d92b226a94aa59287ddc81c7ff77b -> Peer 2b98ac09d43149c5bcfd2c0080ea5379 (127.0.99.133:40367): Couldn't send request to peer 2b98ac09d43149c5bcfd2c0080ea5379. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 18f0be6c9b944ce1b63600f456f272dd. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:58:41.060400 2791 consensus_queue.cc:237] T 149f1f36b61a4d148c2003caaa7fce0c P 809d92b226a94aa59287ddc81c7ff77b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "809d92b226a94aa59287ddc81c7ff77b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33451 } attrs { replace: true } } peers { permanent_uuid: "cb0a6f265d7e4362aea53c916ceb49ec" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 43375 } attrs { promote: true } }
I20240501 07:58:41.061483 4605 raft_consensus.cc:2932] T 149f1f36b61a4d148c2003caaa7fce0c P 809d92b226a94aa59287ddc81c7ff77b [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER cb0a6f265d7e4362aea53c916ceb49ec (127.0.99.132) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "809d92b226a94aa59287ddc81c7ff77b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33451 } attrs { replace: true } } peers { permanent_uuid: "cb0a6f265d7e4362aea53c916ceb49ec" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 43375 } attrs { promote: true } } }
I20240501 07:58:41.063983 2649 catalog_manager.cc:5484] T 149f1f36b61a4d148c2003caaa7fce0c P 809d92b226a94aa59287ddc81c7ff77b reported cstate change: config changed from index -1 to 4, NON_VOTER cb0a6f265d7e4362aea53c916ceb49ec (127.0.99.132) added. New cstate: current_term: 1 leader_uuid: "809d92b226a94aa59287ddc81c7ff77b" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "809d92b226a94aa59287ddc81c7ff77b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33451 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cb0a6f265d7e4362aea53c916ceb49ec" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 43375 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240501 07:58:41.065856 2724 consensus_peers.cc:485] T 149f1f36b61a4d148c2003caaa7fce0c P 809d92b226a94aa59287ddc81c7ff77b -> Peer cb0a6f265d7e4362aea53c916ceb49ec (127.0.99.132:43375): Couldn't send request to peer cb0a6f265d7e4362aea53c916ceb49ec. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 149f1f36b61a4d148c2003caaa7fce0c. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:58:41.070550 2922 consensus_queue.cc:237] T c63214bc45b3424492cc114662f6839e P 853826bd6abf488a8895392f3470a546 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "853826bd6abf488a8895392f3470a546" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39801 } attrs { replace: true } } peers { permanent_uuid: "2471afee7ac94728a1506dc907ef8fbb" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 38167 } attrs { promote: true } }
I20240501 07:58:41.071537 4593 raft_consensus.cc:2932] T c63214bc45b3424492cc114662f6839e P 853826bd6abf488a8895392f3470a546 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2471afee7ac94728a1506dc907ef8fbb (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "853826bd6abf488a8895392f3470a546" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39801 } attrs { replace: true } } peers { permanent_uuid: "2471afee7ac94728a1506dc907ef8fbb" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 38167 } attrs { promote: true } } }
I20240501 07:58:41.074303 2649 catalog_manager.cc:5484] T c63214bc45b3424492cc114662f6839e P 853826bd6abf488a8895392f3470a546 reported cstate change: config changed from index -1 to 4, NON_VOTER 2471afee7ac94728a1506dc907ef8fbb (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "853826bd6abf488a8895392f3470a546" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "853826bd6abf488a8895392f3470a546" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39801 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2471afee7ac94728a1506dc907ef8fbb" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 38167 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240501 07:58:41.075018 2858 consensus_peers.cc:485] T c63214bc45b3424492cc114662f6839e P 853826bd6abf488a8895392f3470a546 -> Peer 2471afee7ac94728a1506dc907ef8fbb (127.0.99.134:38167): Couldn't send request to peer 2471afee7ac94728a1506dc907ef8fbb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c63214bc45b3424492cc114662f6839e. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:58:41.081908 2791 consensus_queue.cc:237] T db9747650ebf4556ac30e4138821695b P 809d92b226a94aa59287ddc81c7ff77b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "809d92b226a94aa59287ddc81c7ff77b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33451 } attrs { replace: true } } peers { permanent_uuid: "98526b95c3e94668a1673fe660495876" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39073 } attrs { promote: true } }
I20240501 07:58:41.082866 4605 raft_consensus.cc:2932] T db9747650ebf4556ac30e4138821695b P 809d92b226a94aa59287ddc81c7ff77b [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 98526b95c3e94668a1673fe660495876 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "809d92b226a94aa59287ddc81c7ff77b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33451 } attrs { replace: true } } peers { permanent_uuid: "98526b95c3e94668a1673fe660495876" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39073 } attrs { promote: true } } }
W20240501 07:58:41.083274 2725 consensus_peers.cc:485] T db9747650ebf4556ac30e4138821695b P 809d92b226a94aa59287ddc81c7ff77b -> Peer 98526b95c3e94668a1673fe660495876 (127.0.99.135:39073): Couldn't send request to peer 98526b95c3e94668a1673fe660495876. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: db9747650ebf4556ac30e4138821695b. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:58:41.085209 2649 catalog_manager.cc:5484] T db9747650ebf4556ac30e4138821695b P 809d92b226a94aa59287ddc81c7ff77b reported cstate change: config changed from index -1 to 4, NON_VOTER 98526b95c3e94668a1673fe660495876 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "809d92b226a94aa59287ddc81c7ff77b" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "809d92b226a94aa59287ddc81c7ff77b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33451 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "98526b95c3e94668a1673fe660495876" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39073 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 07:58:41.091651 2922 consensus_queue.cc:237] T ef28ebd14a684f1f9f5d6718bf8284b4 P 853826bd6abf488a8895392f3470a546 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "853826bd6abf488a8895392f3470a546" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39801 } attrs { replace: true } } peers { permanent_uuid: "9846630212094f49930ef629ee885368" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 33467 } attrs { promote: true } }
I20240501 07:58:41.092543 4594 raft_consensus.cc:2932] T ef28ebd14a684f1f9f5d6718bf8284b4 P 853826bd6abf488a8895392f3470a546 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9846630212094f49930ef629ee885368 (127.0.99.131) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "853826bd6abf488a8895392f3470a546" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39801 } attrs { replace: true } } peers { permanent_uuid: "9846630212094f49930ef629ee885368" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 33467 } attrs { promote: true } } }
W20240501 07:58:41.092933 2855 consensus_peers.cc:485] T ef28ebd14a684f1f9f5d6718bf8284b4 P 853826bd6abf488a8895392f3470a546 -> Peer 9846630212094f49930ef629ee885368 (127.0.99.131:33467): Couldn't send request to peer 9846630212094f49930ef629ee885368. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ef28ebd14a684f1f9f5d6718bf8284b4. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:58:41.094753 2649 catalog_manager.cc:5484] T ef28ebd14a684f1f9f5d6718bf8284b4 P 853826bd6abf488a8895392f3470a546 reported cstate change: config changed from index -1 to 4, NON_VOTER 9846630212094f49930ef629ee885368 (127.0.99.131) added. New cstate: current_term: 1 leader_uuid: "853826bd6abf488a8895392f3470a546" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "853826bd6abf488a8895392f3470a546" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39801 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9846630212094f49930ef629ee885368" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 33467 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 07:58:41.387912 4372 heartbeater.cc:503] Master 127.0.99.190:42355 was elected leader, sending a full tablet report...
I20240501 07:58:41.404587 4741 ts_tablet_manager.cc:902] T fa6cf4447cfe49ecbbfe94accb24aa97 P 98526b95c3e94668a1673fe660495876: Initiating tablet copy from peer 809d92b226a94aa59287ddc81c7ff77b (127.0.99.129:33451)
I20240501 07:58:41.405942 4741 tablet_copy_client.cc:308] T fa6cf4447cfe49ecbbfe94accb24aa97 P 98526b95c3e94668a1673fe660495876: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:33451
I20240501 07:58:41.409726 2811 tablet_copy_service.cc:140] P 809d92b226a94aa59287ddc81c7ff77b: Received BeginTabletCopySession request for tablet fa6cf4447cfe49ecbbfe94accb24aa97 from peer 98526b95c3e94668a1673fe660495876 ({username='slave'} at 127.0.99.135:42955)
I20240501 07:58:41.409876 2811 tablet_copy_service.cc:161] P 809d92b226a94aa59287ddc81c7ff77b: Beginning new tablet copy session on tablet fa6cf4447cfe49ecbbfe94accb24aa97 from peer 98526b95c3e94668a1673fe660495876 at {username='slave'} at 127.0.99.135:42955: session id = 98526b95c3e94668a1673fe660495876-fa6cf4447cfe49ecbbfe94accb24aa97
I20240501 07:58:41.411087 2811 tablet_copy_source_session.cc:205] T fa6cf4447cfe49ecbbfe94accb24aa97 P 809d92b226a94aa59287ddc81c7ff77b: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:58:41.414412 4741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa6cf4447cfe49ecbbfe94accb24aa97. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:41.420547 4741 tablet_copy_client.cc:783] T fa6cf4447cfe49ecbbfe94accb24aa97 P 98526b95c3e94668a1673fe660495876: tablet copy: Starting download of 0 data blocks...
I20240501 07:58:41.420801 4741 tablet_copy_client.cc:647] T fa6cf4447cfe49ecbbfe94accb24aa97 P 98526b95c3e94668a1673fe660495876: tablet copy: Starting download of 1 WAL segments...
I20240501 07:58:41.423017 4741 tablet_copy_client.cc:523] T fa6cf4447cfe49ecbbfe94accb24aa97 P 98526b95c3e94668a1673fe660495876: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:58:41.424886 4741 tablet_bootstrap.cc:492] T fa6cf4447cfe49ecbbfe94accb24aa97 P 98526b95c3e94668a1673fe660495876: Bootstrap starting.
I20240501 07:58:41.430513 4741 log.cc:824] T fa6cf4447cfe49ecbbfe94accb24aa97 P 98526b95c3e94668a1673fe660495876: Log is configured to *not* fsync() on all Append() calls
I20240501 07:58:41.450757 4748 ts_tablet_manager.cc:902] T 18f0be6c9b944ce1b63600f456f272dd P 2b98ac09d43149c5bcfd2c0080ea5379: Initiating tablet copy from peer 809d92b226a94aa59287ddc81c7ff77b (127.0.99.129:33451)
I20240501 07:58:41.451613 4748 tablet_copy_client.cc:308] T 18f0be6c9b944ce1b63600f456f272dd P 2b98ac09d43149c5bcfd2c0080ea5379: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:33451
I20240501 07:58:41.451872 4741 tablet_bootstrap.cc:492] T fa6cf4447cfe49ecbbfe94accb24aa97 P 98526b95c3e94668a1673fe660495876: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:58:41.452256 4741 tablet_bootstrap.cc:492] T fa6cf4447cfe49ecbbfe94accb24aa97 P 98526b95c3e94668a1673fe660495876: Bootstrap complete.
I20240501 07:58:41.452620 4741 ts_tablet_manager.cc:1367] T fa6cf4447cfe49ecbbfe94accb24aa97 P 98526b95c3e94668a1673fe660495876: Time spent bootstrapping tablet: real 0.028s user 0.026s sys 0.000s
I20240501 07:58:41.455222 4741 raft_consensus.cc:348] T fa6cf4447cfe49ecbbfe94accb24aa97 P 98526b95c3e94668a1673fe660495876 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "809d92b226a94aa59287ddc81c7ff77b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33451 } attrs { replace: true } } peers { permanent_uuid: "98526b95c3e94668a1673fe660495876" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39073 } attrs { promote: true } }
I20240501 07:58:41.455547 4741 raft_consensus.cc:729] T fa6cf4447cfe49ecbbfe94accb24aa97 P 98526b95c3e94668a1673fe660495876 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 98526b95c3e94668a1673fe660495876, State: Initialized, Role: LEARNER
I20240501 07:58:41.455925 2811 tablet_copy_service.cc:140] P 809d92b226a94aa59287ddc81c7ff77b: Received BeginTabletCopySession request for tablet 18f0be6c9b944ce1b63600f456f272dd from peer 2b98ac09d43149c5bcfd2c0080ea5379 ({username='slave'} at 127.0.99.133:42617)
I20240501 07:58:41.455932 4741 consensus_queue.cc:260] T fa6cf4447cfe49ecbbfe94accb24aa97 P 98526b95c3e94668a1673fe660495876 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "809d92b226a94aa59287ddc81c7ff77b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33451 } attrs { replace: true } } peers { permanent_uuid: "98526b95c3e94668a1673fe660495876" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39073 } attrs { promote: true } }
I20240501 07:58:41.456064 2811 tablet_copy_service.cc:161] P 809d92b226a94aa59287ddc81c7ff77b: Beginning new tablet copy session on tablet 18f0be6c9b944ce1b63600f456f272dd from peer 2b98ac09d43149c5bcfd2c0080ea5379 at {username='slave'} at 127.0.99.133:42617: session id = 2b98ac09d43149c5bcfd2c0080ea5379-18f0be6c9b944ce1b63600f456f272dd
I20240501 07:58:41.456646 4503 heartbeater.cc:503] Master 127.0.99.190:42355 was elected leader, sending a full tablet report...
I20240501 07:58:41.456941 4741 ts_tablet_manager.cc:1398] T fa6cf4447cfe49ecbbfe94accb24aa97 P 98526b95c3e94668a1673fe660495876: Time spent starting tablet: real 0.004s user 0.007s sys 0.000s
I20240501 07:58:41.457093 2811 tablet_copy_source_session.cc:205] T 18f0be6c9b944ce1b63600f456f272dd P 809d92b226a94aa59287ddc81c7ff77b: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:58:41.457844 2811 tablet_copy_service.cc:342] P 809d92b226a94aa59287ddc81c7ff77b: Request end of tablet copy session 98526b95c3e94668a1673fe660495876-fa6cf4447cfe49ecbbfe94accb24aa97 received from {username='slave'} at 127.0.99.135:42955
I20240501 07:58:41.457944 2811 tablet_copy_service.cc:433] P 809d92b226a94aa59287ddc81c7ff77b: ending tablet copy session 98526b95c3e94668a1673fe660495876-fa6cf4447cfe49ecbbfe94accb24aa97 on tablet fa6cf4447cfe49ecbbfe94accb24aa97 with peer 98526b95c3e94668a1673fe660495876
I20240501 07:58:41.459043 4748 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18f0be6c9b944ce1b63600f456f272dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:41.462275 4748 tablet_copy_client.cc:783] T 18f0be6c9b944ce1b63600f456f272dd P 2b98ac09d43149c5bcfd2c0080ea5379: tablet copy: Starting download of 0 data blocks...
I20240501 07:58:41.462460 4748 tablet_copy_client.cc:647] T 18f0be6c9b944ce1b63600f456f272dd P 2b98ac09d43149c5bcfd2c0080ea5379: tablet copy: Starting download of 1 WAL segments...
I20240501 07:58:41.463933 4748 tablet_copy_client.cc:523] T 18f0be6c9b944ce1b63600f456f272dd P 2b98ac09d43149c5bcfd2c0080ea5379: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:58:41.465219 4748 tablet_bootstrap.cc:492] T 18f0be6c9b944ce1b63600f456f272dd P 2b98ac09d43149c5bcfd2c0080ea5379: Bootstrap starting.
I20240501 07:58:41.487002 4748 tablet_bootstrap.cc:492] T 18f0be6c9b944ce1b63600f456f272dd P 2b98ac09d43149c5bcfd2c0080ea5379: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:58:41.487325 4748 tablet_bootstrap.cc:492] T 18f0be6c9b944ce1b63600f456f272dd P 2b98ac09d43149c5bcfd2c0080ea5379: Bootstrap complete.
I20240501 07:58:41.487478 4748 ts_tablet_manager.cc:1367] T 18f0be6c9b944ce1b63600f456f272dd P 2b98ac09d43149c5bcfd2c0080ea5379: Time spent bootstrapping tablet: real 0.022s user 0.020s sys 0.000s
I20240501 07:58:41.488001 4748 raft_consensus.cc:348] T 18f0be6c9b944ce1b63600f456f272dd P 2b98ac09d43149c5bcfd2c0080ea5379 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "809d92b226a94aa59287ddc81c7ff77b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33451 } attrs { replace: true } } peers { permanent_uuid: "2b98ac09d43149c5bcfd2c0080ea5379" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 40367 } attrs { promote: true } }
I20240501 07:58:41.488108 4748 raft_consensus.cc:729] T 18f0be6c9b944ce1b63600f456f272dd P 2b98ac09d43149c5bcfd2c0080ea5379 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2b98ac09d43149c5bcfd2c0080ea5379, State: Initialized, Role: LEARNER
I20240501 07:58:41.488198 4748 consensus_queue.cc:260] T 18f0be6c9b944ce1b63600f456f272dd P 2b98ac09d43149c5bcfd2c0080ea5379 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "809d92b226a94aa59287ddc81c7ff77b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33451 } attrs { replace: true } } peers { permanent_uuid: "2b98ac09d43149c5bcfd2c0080ea5379" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 40367 } attrs { promote: true } }
I20240501 07:58:41.488554 4748 ts_tablet_manager.cc:1398] T 18f0be6c9b944ce1b63600f456f272dd P 2b98ac09d43149c5bcfd2c0080ea5379: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20240501 07:58:41.489298 2811 tablet_copy_service.cc:342] P 809d92b226a94aa59287ddc81c7ff77b: Request end of tablet copy session 2b98ac09d43149c5bcfd2c0080ea5379-18f0be6c9b944ce1b63600f456f272dd received from {username='slave'} at 127.0.99.133:42617
I20240501 07:58:41.489392 2811 tablet_copy_service.cc:433] P 809d92b226a94aa59287ddc81c7ff77b: ending tablet copy session 2b98ac09d43149c5bcfd2c0080ea5379-18f0be6c9b944ce1b63600f456f272dd on tablet 18f0be6c9b944ce1b63600f456f272dd with peer 2b98ac09d43149c5bcfd2c0080ea5379
I20240501 07:58:41.545992 4757 ts_tablet_manager.cc:902] T ef28ebd14a684f1f9f5d6718bf8284b4 P 9846630212094f49930ef629ee885368: Initiating tablet copy from peer 853826bd6abf488a8895392f3470a546 (127.0.99.130:39801)
I20240501 07:58:41.546677 4757 tablet_copy_client.cc:308] T ef28ebd14a684f1f9f5d6718bf8284b4 P 9846630212094f49930ef629ee885368: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:39801
I20240501 07:58:41.550665 2942 tablet_copy_service.cc:140] P 853826bd6abf488a8895392f3470a546: Received BeginTabletCopySession request for tablet ef28ebd14a684f1f9f5d6718bf8284b4 from peer 9846630212094f49930ef629ee885368 ({username='slave'} at 127.0.99.131:41649)
I20240501 07:58:41.550828 2942 tablet_copy_service.cc:161] P 853826bd6abf488a8895392f3470a546: Beginning new tablet copy session on tablet ef28ebd14a684f1f9f5d6718bf8284b4 from peer 9846630212094f49930ef629ee885368 at {username='slave'} at 127.0.99.131:41649: session id = 9846630212094f49930ef629ee885368-ef28ebd14a684f1f9f5d6718bf8284b4
I20240501 07:58:41.552109 2942 tablet_copy_source_session.cc:205] T ef28ebd14a684f1f9f5d6718bf8284b4 P 853826bd6abf488a8895392f3470a546: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:58:41.555557 4757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef28ebd14a684f1f9f5d6718bf8284b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:41.560812 4763 ts_tablet_manager.cc:902] T 102891a28f95444dac4798b2aace3857 P 9846630212094f49930ef629ee885368: Initiating tablet copy from peer 853826bd6abf488a8895392f3470a546 (127.0.99.130:39801)
I20240501 07:58:41.561367 4757 tablet_copy_client.cc:783] T ef28ebd14a684f1f9f5d6718bf8284b4 P 9846630212094f49930ef629ee885368: tablet copy: Starting download of 0 data blocks...
I20240501 07:58:41.561549 4763 tablet_copy_client.cc:308] T 102891a28f95444dac4798b2aace3857 P 9846630212094f49930ef629ee885368: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:39801
I20240501 07:58:41.561568 4757 tablet_copy_client.cc:647] T ef28ebd14a684f1f9f5d6718bf8284b4 P 9846630212094f49930ef629ee885368: tablet copy: Starting download of 1 WAL segments...
I20240501 07:58:41.562093 2942 tablet_copy_service.cc:140] P 853826bd6abf488a8895392f3470a546: Received BeginTabletCopySession request for tablet 102891a28f95444dac4798b2aace3857 from peer 9846630212094f49930ef629ee885368 ({username='slave'} at 127.0.99.131:41649)
I20240501 07:58:41.562230 2942 tablet_copy_service.cc:161] P 853826bd6abf488a8895392f3470a546: Beginning new tablet copy session on tablet 102891a28f95444dac4798b2aace3857 from peer 9846630212094f49930ef629ee885368 at {username='slave'} at 127.0.99.131:41649: session id = 9846630212094f49930ef629ee885368-102891a28f95444dac4798b2aace3857
I20240501 07:58:41.563212 2942 tablet_copy_source_session.cc:205] T 102891a28f95444dac4798b2aace3857 P 853826bd6abf488a8895392f3470a546: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:58:41.563539 4757 tablet_copy_client.cc:523] T ef28ebd14a684f1f9f5d6718bf8284b4 P 9846630212094f49930ef629ee885368: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:58:41.564036 4763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 102891a28f95444dac4798b2aace3857. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:41.565349 4757 tablet_bootstrap.cc:492] T ef28ebd14a684f1f9f5d6718bf8284b4 P 9846630212094f49930ef629ee885368: Bootstrap starting.
I20240501 07:58:41.566123 4763 tablet_copy_client.cc:783] T 102891a28f95444dac4798b2aace3857 P 9846630212094f49930ef629ee885368: tablet copy: Starting download of 0 data blocks...
I20240501 07:58:41.566264 4763 tablet_copy_client.cc:647] T 102891a28f95444dac4798b2aace3857 P 9846630212094f49930ef629ee885368: tablet copy: Starting download of 1 WAL segments...
I20240501 07:58:41.570325 4763 tablet_copy_client.cc:523] T 102891a28f95444dac4798b2aace3857 P 9846630212094f49930ef629ee885368: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:58:41.571712 4763 tablet_bootstrap.cc:492] T 102891a28f95444dac4798b2aace3857 P 9846630212094f49930ef629ee885368: Bootstrap starting.
I20240501 07:58:41.572093 4757 log.cc:824] T ef28ebd14a684f1f9f5d6718bf8284b4 P 9846630212094f49930ef629ee885368: Log is configured to *not* fsync() on all Append() calls
I20240501 07:58:41.592449 4757 tablet_bootstrap.cc:492] T ef28ebd14a684f1f9f5d6718bf8284b4 P 9846630212094f49930ef629ee885368: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:58:41.592449 4763 tablet_bootstrap.cc:492] T 102891a28f95444dac4798b2aace3857 P 9846630212094f49930ef629ee885368: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:58:41.592855 4757 tablet_bootstrap.cc:492] T ef28ebd14a684f1f9f5d6718bf8284b4 P 9846630212094f49930ef629ee885368: Bootstrap complete.
I20240501 07:58:41.592857 4763 tablet_bootstrap.cc:492] T 102891a28f95444dac4798b2aace3857 P 9846630212094f49930ef629ee885368: Bootstrap complete.
I20240501 07:58:41.593212 4763 ts_tablet_manager.cc:1367] T 102891a28f95444dac4798b2aace3857 P 9846630212094f49930ef629ee885368: Time spent bootstrapping tablet: real 0.022s user 0.022s sys 0.000s
I20240501 07:58:41.593212 4757 ts_tablet_manager.cc:1367] T ef28ebd14a684f1f9f5d6718bf8284b4 P 9846630212094f49930ef629ee885368: Time spent bootstrapping tablet: real 0.028s user 0.027s sys 0.000s
I20240501 07:58:41.595288 4763 raft_consensus.cc:348] T 102891a28f95444dac4798b2aace3857 P 9846630212094f49930ef629ee885368 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "853826bd6abf488a8895392f3470a546" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39801 } attrs { replace: true } } peers { permanent_uuid: "9846630212094f49930ef629ee885368" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 33467 } attrs { promote: true } }
I20240501 07:58:41.595288 4757 raft_consensus.cc:348] T ef28ebd14a684f1f9f5d6718bf8284b4 P 9846630212094f49930ef629ee885368 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "853826bd6abf488a8895392f3470a546" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39801 } attrs { replace: true } } peers { permanent_uuid: "9846630212094f49930ef629ee885368" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 33467 } attrs { promote: true } }
I20240501 07:58:41.595532 4757 raft_consensus.cc:729] T ef28ebd14a684f1f9f5d6718bf8284b4 P 9846630212094f49930ef629ee885368 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9846630212094f49930ef629ee885368, State: Initialized, Role: LEARNER
I20240501 07:58:41.595532 4763 raft_consensus.cc:729] T 102891a28f95444dac4798b2aace3857 P 9846630212094f49930ef629ee885368 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9846630212094f49930ef629ee885368, State: Initialized, Role: LEARNER
I20240501 07:58:41.595882 4763 consensus_queue.cc:260] T 102891a28f95444dac4798b2aace3857 P 9846630212094f49930ef629ee885368 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "853826bd6abf488a8895392f3470a546" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39801 } attrs { replace: true } } peers { permanent_uuid: "9846630212094f49930ef629ee885368" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 33467 } attrs { promote: true } }
I20240501 07:58:41.595883 4757 consensus_queue.cc:260] T ef28ebd14a684f1f9f5d6718bf8284b4 P 9846630212094f49930ef629ee885368 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "853826bd6abf488a8895392f3470a546" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39801 } attrs { replace: true } } peers { permanent_uuid: "9846630212094f49930ef629ee885368" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 33467 } attrs { promote: true } }
I20240501 07:58:41.596993 4757 ts_tablet_manager.cc:1398] T ef28ebd14a684f1f9f5d6718bf8284b4 P 9846630212094f49930ef629ee885368: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20240501 07:58:41.597085 4763 ts_tablet_manager.cc:1398] T 102891a28f95444dac4798b2aace3857 P 9846630212094f49930ef629ee885368: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20240501 07:58:41.597781 4772 ts_tablet_manager.cc:902] T 0b2690f2189b43f9a45e1a90605623ad P cb0a6f265d7e4362aea53c916ceb49ec: Initiating tablet copy from peer 853826bd6abf488a8895392f3470a546 (127.0.99.130:39801)
I20240501 07:58:41.597877 2942 tablet_copy_service.cc:342] P 853826bd6abf488a8895392f3470a546: Request end of tablet copy session 9846630212094f49930ef629ee885368-102891a28f95444dac4798b2aace3857 received from {username='slave'} at 127.0.99.131:41649
I20240501 07:58:41.597959 2942 tablet_copy_service.cc:433] P 853826bd6abf488a8895392f3470a546: ending tablet copy session 9846630212094f49930ef629ee885368-102891a28f95444dac4798b2aace3857 on tablet 102891a28f95444dac4798b2aace3857 with peer 9846630212094f49930ef629ee885368
I20240501 07:58:41.598217 2941 tablet_copy_service.cc:342] P 853826bd6abf488a8895392f3470a546: Request end of tablet copy session 9846630212094f49930ef629ee885368-ef28ebd14a684f1f9f5d6718bf8284b4 received from {username='slave'} at 127.0.99.131:41649
I20240501 07:58:41.598297 2941 tablet_copy_service.cc:433] P 853826bd6abf488a8895392f3470a546: ending tablet copy session 9846630212094f49930ef629ee885368-ef28ebd14a684f1f9f5d6718bf8284b4 on tablet ef28ebd14a684f1f9f5d6718bf8284b4 with peer 9846630212094f49930ef629ee885368
I20240501 07:58:41.598513 4772 tablet_copy_client.cc:308] T 0b2690f2189b43f9a45e1a90605623ad P cb0a6f265d7e4362aea53c916ceb49ec: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:39801
I20240501 07:58:41.600536 4778 ts_tablet_manager.cc:902] T 149f1f36b61a4d148c2003caaa7fce0c P cb0a6f265d7e4362aea53c916ceb49ec: Initiating tablet copy from peer 809d92b226a94aa59287ddc81c7ff77b (127.0.99.129:33451)
I20240501 07:58:41.601127 4778 tablet_copy_client.cc:308] T 149f1f36b61a4d148c2003caaa7fce0c P cb0a6f265d7e4362aea53c916ceb49ec: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:33451
I20240501 07:58:41.603310 2941 tablet_copy_service.cc:140] P 853826bd6abf488a8895392f3470a546: Received BeginTabletCopySession request for tablet 0b2690f2189b43f9a45e1a90605623ad from peer cb0a6f265d7e4362aea53c916ceb49ec ({username='slave'} at 127.0.99.132:42449)
I20240501 07:58:41.603403 2941 tablet_copy_service.cc:161] P 853826bd6abf488a8895392f3470a546: Beginning new tablet copy session on tablet 0b2690f2189b43f9a45e1a90605623ad from peer cb0a6f265d7e4362aea53c916ceb49ec at {username='slave'} at 127.0.99.132:42449: session id = cb0a6f265d7e4362aea53c916ceb49ec-0b2690f2189b43f9a45e1a90605623ad
I20240501 07:58:41.604264 2941 tablet_copy_source_session.cc:205] T 0b2690f2189b43f9a45e1a90605623ad P 853826bd6abf488a8895392f3470a546: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:58:41.604773 2811 tablet_copy_service.cc:140] P 809d92b226a94aa59287ddc81c7ff77b: Received BeginTabletCopySession request for tablet 149f1f36b61a4d148c2003caaa7fce0c from peer cb0a6f265d7e4362aea53c916ceb49ec ({username='slave'} at 127.0.99.132:36549)
I20240501 07:58:41.604900 2811 tablet_copy_service.cc:161] P 809d92b226a94aa59287ddc81c7ff77b: Beginning new tablet copy session on tablet 149f1f36b61a4d148c2003caaa7fce0c from peer cb0a6f265d7e4362aea53c916ceb49ec at {username='slave'} at 127.0.99.132:36549: session id = cb0a6f265d7e4362aea53c916ceb49ec-149f1f36b61a4d148c2003caaa7fce0c
I20240501 07:58:41.605613 2811 tablet_copy_source_session.cc:205] T 149f1f36b61a4d148c2003caaa7fce0c P 809d92b226a94aa59287ddc81c7ff77b: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:58:41.607630 4778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 149f1f36b61a4d148c2003caaa7fce0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:41.608093 4772 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b2690f2189b43f9a45e1a90605623ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:41.614604 4772 tablet_copy_client.cc:783] T 0b2690f2189b43f9a45e1a90605623ad P cb0a6f265d7e4362aea53c916ceb49ec: tablet copy: Starting download of 0 data blocks...
I20240501 07:58:41.614678 4778 tablet_copy_client.cc:783] T 149f1f36b61a4d148c2003caaa7fce0c P cb0a6f265d7e4362aea53c916ceb49ec: tablet copy: Starting download of 0 data blocks...
I20240501 07:58:41.614816 4772 tablet_copy_client.cc:647] T 0b2690f2189b43f9a45e1a90605623ad P cb0a6f265d7e4362aea53c916ceb49ec: tablet copy: Starting download of 1 WAL segments...
I20240501 07:58:41.614888 4778 tablet_copy_client.cc:647] T 149f1f36b61a4d148c2003caaa7fce0c P cb0a6f265d7e4362aea53c916ceb49ec: tablet copy: Starting download of 1 WAL segments...
I20240501 07:58:41.616547 4778 tablet_copy_client.cc:523] T 149f1f36b61a4d148c2003caaa7fce0c P cb0a6f265d7e4362aea53c916ceb49ec: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:58:41.616547 4772 tablet_copy_client.cc:523] T 0b2690f2189b43f9a45e1a90605623ad P cb0a6f265d7e4362aea53c916ceb49ec: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:58:41.618495 4772 tablet_bootstrap.cc:492] T 0b2690f2189b43f9a45e1a90605623ad P cb0a6f265d7e4362aea53c916ceb49ec: Bootstrap starting.
I20240501 07:58:41.618495 4778 tablet_bootstrap.cc:492] T 149f1f36b61a4d148c2003caaa7fce0c P cb0a6f265d7e4362aea53c916ceb49ec: Bootstrap starting.
I20240501 07:58:41.625077 4778 log.cc:824] T 149f1f36b61a4d148c2003caaa7fce0c P cb0a6f265d7e4362aea53c916ceb49ec: Log is configured to *not* fsync() on all Append() calls
I20240501 07:58:41.636819 4806 ts_tablet_manager.cc:902] T c64dae3606e049e2a72b00148fffc748 P 2471afee7ac94728a1506dc907ef8fbb: Initiating tablet copy from peer 809d92b226a94aa59287ddc81c7ff77b (127.0.99.129:33451)
I20240501 07:58:41.637625 4806 tablet_copy_client.cc:308] T c64dae3606e049e2a72b00148fffc748 P 2471afee7ac94728a1506dc907ef8fbb: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:33451
I20240501 07:58:41.643168 2811 tablet_copy_service.cc:140] P 809d92b226a94aa59287ddc81c7ff77b: Received BeginTabletCopySession request for tablet c64dae3606e049e2a72b00148fffc748 from peer 2471afee7ac94728a1506dc907ef8fbb ({username='slave'} at 127.0.99.134:59109)
I20240501 07:58:41.643292 2811 tablet_copy_service.cc:161] P 809d92b226a94aa59287ddc81c7ff77b: Beginning new tablet copy session on tablet c64dae3606e049e2a72b00148fffc748 from peer 2471afee7ac94728a1506dc907ef8fbb at {username='slave'} at 127.0.99.134:59109: session id = 2471afee7ac94728a1506dc907ef8fbb-c64dae3606e049e2a72b00148fffc748
I20240501 07:58:41.644016 2811 tablet_copy_source_session.cc:205] T c64dae3606e049e2a72b00148fffc748 P 809d92b226a94aa59287ddc81c7ff77b: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:58:41.647832 4806 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c64dae3606e049e2a72b00148fffc748. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:41.655700 4806 tablet_copy_client.cc:783] T c64dae3606e049e2a72b00148fffc748 P 2471afee7ac94728a1506dc907ef8fbb: tablet copy: Starting download of 0 data blocks...
I20240501 07:58:41.655920 4806 tablet_copy_client.cc:647] T c64dae3606e049e2a72b00148fffc748 P 2471afee7ac94728a1506dc907ef8fbb: tablet copy: Starting download of 1 WAL segments...
I20240501 07:58:41.657562 4806 tablet_copy_client.cc:523] T c64dae3606e049e2a72b00148fffc748 P 2471afee7ac94728a1506dc907ef8fbb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:58:41.658298 4778 tablet_bootstrap.cc:492] T 149f1f36b61a4d148c2003caaa7fce0c P cb0a6f265d7e4362aea53c916ceb49ec: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:58:41.658298 4772 tablet_bootstrap.cc:492] T 0b2690f2189b43f9a45e1a90605623ad P cb0a6f265d7e4362aea53c916ceb49ec: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:58:41.658716 4772 tablet_bootstrap.cc:492] T 0b2690f2189b43f9a45e1a90605623ad P cb0a6f265d7e4362aea53c916ceb49ec: Bootstrap complete.
I20240501 07:58:41.658716 4778 tablet_bootstrap.cc:492] T 149f1f36b61a4d148c2003caaa7fce0c P cb0a6f265d7e4362aea53c916ceb49ec: Bootstrap complete.
I20240501 07:58:41.659083 4778 ts_tablet_manager.cc:1367] T 149f1f36b61a4d148c2003caaa7fce0c P cb0a6f265d7e4362aea53c916ceb49ec: Time spent bootstrapping tablet: real 0.041s user 0.033s sys 0.004s
I20240501 07:58:41.659083 4772 ts_tablet_manager.cc:1367] T 0b2690f2189b43f9a45e1a90605623ad P cb0a6f265d7e4362aea53c916ceb49ec: Time spent bootstrapping tablet: real 0.041s user 0.033s sys 0.004s
I20240501 07:58:41.659754 4806 tablet_bootstrap.cc:492] T c64dae3606e049e2a72b00148fffc748 P 2471afee7ac94728a1506dc907ef8fbb: Bootstrap starting.
I20240501 07:58:41.661705 4778 raft_consensus.cc:348] T 149f1f36b61a4d148c2003caaa7fce0c P cb0a6f265d7e4362aea53c916ceb49ec [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "809d92b226a94aa59287ddc81c7ff77b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33451 } attrs { replace: true } } peers { permanent_uuid: "cb0a6f265d7e4362aea53c916ceb49ec" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 43375 } attrs { promote: true } }
I20240501 07:58:41.661705 4772 raft_consensus.cc:348] T 0b2690f2189b43f9a45e1a90605623ad P cb0a6f265d7e4362aea53c916ceb49ec [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "853826bd6abf488a8895392f3470a546" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39801 } attrs { replace: true } } peers { permanent_uuid: "cb0a6f265d7e4362aea53c916ceb49ec" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 43375 } attrs { promote: true } }
I20240501 07:58:41.662000 4778 raft_consensus.cc:729] T 149f1f36b61a4d148c2003caaa7fce0c P cb0a6f265d7e4362aea53c916ceb49ec [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: cb0a6f265d7e4362aea53c916ceb49ec, State: Initialized, Role: LEARNER
I20240501 07:58:41.662034 4772 raft_consensus.cc:729] T 0b2690f2189b43f9a45e1a90605623ad P cb0a6f265d7e4362aea53c916ceb49ec [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: cb0a6f265d7e4362aea53c916ceb49ec, State: Initialized, Role: LEARNER
I20240501 07:58:41.662376 4772 consensus_queue.cc:260] T 0b2690f2189b43f9a45e1a90605623ad P cb0a6f265d7e4362aea53c916ceb49ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "853826bd6abf488a8895392f3470a546" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39801 } attrs { replace: true } } peers { permanent_uuid: "cb0a6f265d7e4362aea53c916ceb49ec" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 43375 } attrs { promote: true } }
I20240501 07:58:41.662588 4778 consensus_queue.cc:260] T 149f1f36b61a4d148c2003caaa7fce0c P cb0a6f265d7e4362aea53c916ceb49ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "809d92b226a94aa59287ddc81c7ff77b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33451 } attrs { replace: true } } peers { permanent_uuid: "cb0a6f265d7e4362aea53c916ceb49ec" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 43375 } attrs { promote: true } }
I20240501 07:58:41.663264 4778 ts_tablet_manager.cc:1398] T 149f1f36b61a4d148c2003caaa7fce0c P cb0a6f265d7e4362aea53c916ceb49ec: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20240501 07:58:41.663264 4772 ts_tablet_manager.cc:1398] T 0b2690f2189b43f9a45e1a90605623ad P cb0a6f265d7e4362aea53c916ceb49ec: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 07:58:41.664098 2941 tablet_copy_service.cc:342] P 853826bd6abf488a8895392f3470a546: Request end of tablet copy session cb0a6f265d7e4362aea53c916ceb49ec-0b2690f2189b43f9a45e1a90605623ad received from {username='slave'} at 127.0.99.132:42449
I20240501 07:58:41.664113 2811 tablet_copy_service.cc:342] P 809d92b226a94aa59287ddc81c7ff77b: Request end of tablet copy session cb0a6f265d7e4362aea53c916ceb49ec-149f1f36b61a4d148c2003caaa7fce0c received from {username='slave'} at 127.0.99.132:36549
I20240501 07:58:41.664181 2941 tablet_copy_service.cc:433] P 853826bd6abf488a8895392f3470a546: ending tablet copy session cb0a6f265d7e4362aea53c916ceb49ec-0b2690f2189b43f9a45e1a90605623ad on tablet 0b2690f2189b43f9a45e1a90605623ad with peer cb0a6f265d7e4362aea53c916ceb49ec
I20240501 07:58:41.664194 2811 tablet_copy_service.cc:433] P 809d92b226a94aa59287ddc81c7ff77b: ending tablet copy session cb0a6f265d7e4362aea53c916ceb49ec-149f1f36b61a4d148c2003caaa7fce0c on tablet 149f1f36b61a4d148c2003caaa7fce0c with peer cb0a6f265d7e4362aea53c916ceb49ec
I20240501 07:58:41.668047 4806 log.cc:824] T c64dae3606e049e2a72b00148fffc748 P 2471afee7ac94728a1506dc907ef8fbb: Log is configured to *not* fsync() on all Append() calls
I20240501 07:58:41.672812 4836 ts_tablet_manager.cc:902] T c63214bc45b3424492cc114662f6839e P 2471afee7ac94728a1506dc907ef8fbb: Initiating tablet copy from peer 853826bd6abf488a8895392f3470a546 (127.0.99.130:39801)
I20240501 07:58:41.673388 4836 tablet_copy_client.cc:308] T c63214bc45b3424492cc114662f6839e P 2471afee7ac94728a1506dc907ef8fbb: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:39801
I20240501 07:58:41.678190 2941 tablet_copy_service.cc:140] P 853826bd6abf488a8895392f3470a546: Received BeginTabletCopySession request for tablet c63214bc45b3424492cc114662f6839e from peer 2471afee7ac94728a1506dc907ef8fbb ({username='slave'} at 127.0.99.134:44449)
I20240501 07:58:41.678333 2941 tablet_copy_service.cc:161] P 853826bd6abf488a8895392f3470a546: Beginning new tablet copy session on tablet c63214bc45b3424492cc114662f6839e from peer 2471afee7ac94728a1506dc907ef8fbb at {username='slave'} at 127.0.99.134:44449: session id = 2471afee7ac94728a1506dc907ef8fbb-c63214bc45b3424492cc114662f6839e
I20240501 07:58:41.679244 2941 tablet_copy_source_session.cc:205] T c63214bc45b3424492cc114662f6839e P 853826bd6abf488a8895392f3470a546: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:58:41.680151 4836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c63214bc45b3424492cc114662f6839e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:41.682006 4836 tablet_copy_client.cc:783] T c63214bc45b3424492cc114662f6839e P 2471afee7ac94728a1506dc907ef8fbb: tablet copy: Starting download of 0 data blocks...
I20240501 07:58:41.682160 4836 tablet_copy_client.cc:647] T c63214bc45b3424492cc114662f6839e P 2471afee7ac94728a1506dc907ef8fbb: tablet copy: Starting download of 1 WAL segments...
I20240501 07:58:41.683259 4836 tablet_copy_client.cc:523] T c63214bc45b3424492cc114662f6839e P 2471afee7ac94728a1506dc907ef8fbb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:58:41.684298 4836 tablet_bootstrap.cc:492] T c63214bc45b3424492cc114662f6839e P 2471afee7ac94728a1506dc907ef8fbb: Bootstrap starting.
I20240501 07:58:41.692168 4806 tablet_bootstrap.cc:492] T c64dae3606e049e2a72b00148fffc748 P 2471afee7ac94728a1506dc907ef8fbb: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:58:41.692533 4806 tablet_bootstrap.cc:492] T c64dae3606e049e2a72b00148fffc748 P 2471afee7ac94728a1506dc907ef8fbb: Bootstrap complete.
I20240501 07:58:41.692832 4806 ts_tablet_manager.cc:1367] T c64dae3606e049e2a72b00148fffc748 P 2471afee7ac94728a1506dc907ef8fbb: Time spent bootstrapping tablet: real 0.033s user 0.023s sys 0.008s
I20240501 07:58:41.695108 4806 raft_consensus.cc:348] T c64dae3606e049e2a72b00148fffc748 P 2471afee7ac94728a1506dc907ef8fbb [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "809d92b226a94aa59287ddc81c7ff77b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33451 } attrs { replace: true } } peers { permanent_uuid: "2471afee7ac94728a1506dc907ef8fbb" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 38167 } attrs { promote: true } }
I20240501 07:58:41.695320 4806 raft_consensus.cc:729] T c64dae3606e049e2a72b00148fffc748 P 2471afee7ac94728a1506dc907ef8fbb [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2471afee7ac94728a1506dc907ef8fbb, State: Initialized, Role: LEARNER
I20240501 07:58:41.695623 4806 consensus_queue.cc:260] T c64dae3606e049e2a72b00148fffc748 P 2471afee7ac94728a1506dc907ef8fbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "809d92b226a94aa59287ddc81c7ff77b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33451 } attrs { replace: true } } peers { permanent_uuid: "2471afee7ac94728a1506dc907ef8fbb" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 38167 } attrs { promote: true } }
I20240501 07:58:41.696460 4806 ts_tablet_manager.cc:1398] T c64dae3606e049e2a72b00148fffc748 P 2471afee7ac94728a1506dc907ef8fbb: Time spent starting tablet: real 0.004s user 0.006s sys 0.001s
I20240501 07:58:41.697252 2811 tablet_copy_service.cc:342] P 809d92b226a94aa59287ddc81c7ff77b: Request end of tablet copy session 2471afee7ac94728a1506dc907ef8fbb-c64dae3606e049e2a72b00148fffc748 received from {username='slave'} at 127.0.99.134:59109
I20240501 07:58:41.697350 2811 tablet_copy_service.cc:433] P 809d92b226a94aa59287ddc81c7ff77b: ending tablet copy session 2471afee7ac94728a1506dc907ef8fbb-c64dae3606e049e2a72b00148fffc748 on tablet c64dae3606e049e2a72b00148fffc748 with peer 2471afee7ac94728a1506dc907ef8fbb
I20240501 07:58:41.702505 4741 ts_tablet_manager.cc:902] T db9747650ebf4556ac30e4138821695b P 98526b95c3e94668a1673fe660495876: Initiating tablet copy from peer 809d92b226a94aa59287ddc81c7ff77b (127.0.99.129:33451)
I20240501 07:58:41.703011 4741 tablet_copy_client.cc:308] T db9747650ebf4556ac30e4138821695b P 98526b95c3e94668a1673fe660495876: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:33451
I20240501 07:58:41.703465 4836 tablet_bootstrap.cc:492] T c63214bc45b3424492cc114662f6839e P 2471afee7ac94728a1506dc907ef8fbb: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:58:41.703581 2811 tablet_copy_service.cc:140] P 809d92b226a94aa59287ddc81c7ff77b: Received BeginTabletCopySession request for tablet db9747650ebf4556ac30e4138821695b from peer 98526b95c3e94668a1673fe660495876 ({username='slave'} at 127.0.99.135:42955)
I20240501 07:58:41.703698 2811 tablet_copy_service.cc:161] P 809d92b226a94aa59287ddc81c7ff77b: Beginning new tablet copy session on tablet db9747650ebf4556ac30e4138821695b from peer 98526b95c3e94668a1673fe660495876 at {username='slave'} at 127.0.99.135:42955: session id = 98526b95c3e94668a1673fe660495876-db9747650ebf4556ac30e4138821695b
I20240501 07:58:41.703802 4836 tablet_bootstrap.cc:492] T c63214bc45b3424492cc114662f6839e P 2471afee7ac94728a1506dc907ef8fbb: Bootstrap complete.
I20240501 07:58:41.703991 4836 ts_tablet_manager.cc:1367] T c63214bc45b3424492cc114662f6839e P 2471afee7ac94728a1506dc907ef8fbb: Time spent bootstrapping tablet: real 0.020s user 0.017s sys 0.000s
I20240501 07:58:41.704551 2811 tablet_copy_source_session.cc:205] T db9747650ebf4556ac30e4138821695b P 809d92b226a94aa59287ddc81c7ff77b: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:58:41.704524 4836 raft_consensus.cc:348] T c63214bc45b3424492cc114662f6839e P 2471afee7ac94728a1506dc907ef8fbb [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "853826bd6abf488a8895392f3470a546" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39801 } attrs { replace: true } } peers { permanent_uuid: "2471afee7ac94728a1506dc907ef8fbb" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 38167 } attrs { promote: true } }
I20240501 07:58:41.704617 4836 raft_consensus.cc:729] T c63214bc45b3424492cc114662f6839e P 2471afee7ac94728a1506dc907ef8fbb [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2471afee7ac94728a1506dc907ef8fbb, State: Initialized, Role: LEARNER
I20240501 07:58:41.704711 4836 consensus_queue.cc:260] T c63214bc45b3424492cc114662f6839e P 2471afee7ac94728a1506dc907ef8fbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "853826bd6abf488a8895392f3470a546" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39801 } attrs { replace: true } } peers { permanent_uuid: "2471afee7ac94728a1506dc907ef8fbb" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 38167 } attrs { promote: true } }
I20240501 07:58:41.705049 4836 ts_tablet_manager.cc:1398] T c63214bc45b3424492cc114662f6839e P 2471afee7ac94728a1506dc907ef8fbb: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240501 07:58:41.705513 4741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db9747650ebf4556ac30e4138821695b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:41.705653 2941 tablet_copy_service.cc:342] P 853826bd6abf488a8895392f3470a546: Request end of tablet copy session 2471afee7ac94728a1506dc907ef8fbb-c63214bc45b3424492cc114662f6839e received from {username='slave'} at 127.0.99.134:44449
I20240501 07:58:41.705727 2941 tablet_copy_service.cc:433] P 853826bd6abf488a8895392f3470a546: ending tablet copy session 2471afee7ac94728a1506dc907ef8fbb-c63214bc45b3424492cc114662f6839e on tablet c63214bc45b3424492cc114662f6839e with peer 2471afee7ac94728a1506dc907ef8fbb
I20240501 07:58:41.707687 4741 tablet_copy_client.cc:783] T db9747650ebf4556ac30e4138821695b P 98526b95c3e94668a1673fe660495876: tablet copy: Starting download of 0 data blocks...
I20240501 07:58:41.708153 4741 tablet_copy_client.cc:647] T db9747650ebf4556ac30e4138821695b P 98526b95c3e94668a1673fe660495876: tablet copy: Starting download of 1 WAL segments...
I20240501 07:58:41.709357 4741 tablet_copy_client.cc:523] T db9747650ebf4556ac30e4138821695b P 98526b95c3e94668a1673fe660495876: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:58:41.710426 4741 tablet_bootstrap.cc:492] T db9747650ebf4556ac30e4138821695b P 98526b95c3e94668a1673fe660495876: Bootstrap starting.
I20240501 07:58:41.727589 4741 tablet_bootstrap.cc:492] T db9747650ebf4556ac30e4138821695b P 98526b95c3e94668a1673fe660495876: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:58:41.727897 4741 tablet_bootstrap.cc:492] T db9747650ebf4556ac30e4138821695b P 98526b95c3e94668a1673fe660495876: Bootstrap complete.
I20240501 07:58:41.728080 4741 ts_tablet_manager.cc:1367] T db9747650ebf4556ac30e4138821695b P 98526b95c3e94668a1673fe660495876: Time spent bootstrapping tablet: real 0.018s user 0.015s sys 0.000s
I20240501 07:58:41.728596 4741 raft_consensus.cc:348] T db9747650ebf4556ac30e4138821695b P 98526b95c3e94668a1673fe660495876 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "809d92b226a94aa59287ddc81c7ff77b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33451 } attrs { replace: true } } peers { permanent_uuid: "98526b95c3e94668a1673fe660495876" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39073 } attrs { promote: true } }
I20240501 07:58:41.728708 4741 raft_consensus.cc:729] T db9747650ebf4556ac30e4138821695b P 98526b95c3e94668a1673fe660495876 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 98526b95c3e94668a1673fe660495876, State: Initialized, Role: LEARNER
I20240501 07:58:41.728811 4741 consensus_queue.cc:260] T db9747650ebf4556ac30e4138821695b P 98526b95c3e94668a1673fe660495876 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "809d92b226a94aa59287ddc81c7ff77b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33451 } attrs { replace: true } } peers { permanent_uuid: "98526b95c3e94668a1673fe660495876" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39073 } attrs { promote: true } }
I20240501 07:58:41.729112 4741 ts_tablet_manager.cc:1398] T db9747650ebf4556ac30e4138821695b P 98526b95c3e94668a1673fe660495876: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240501 07:58:41.729753 2811 tablet_copy_service.cc:342] P 809d92b226a94aa59287ddc81c7ff77b: Request end of tablet copy session 98526b95c3e94668a1673fe660495876-db9747650ebf4556ac30e4138821695b received from {username='slave'} at 127.0.99.135:42955
I20240501 07:58:41.729840 2811 tablet_copy_service.cc:433] P 809d92b226a94aa59287ddc81c7ff77b: ending tablet copy session 98526b95c3e94668a1673fe660495876-db9747650ebf4556ac30e4138821695b on tablet db9747650ebf4556ac30e4138821695b with peer 98526b95c3e94668a1673fe660495876
I20240501 07:58:41.845293 4186 raft_consensus.cc:1203] T 18f0be6c9b944ce1b63600f456f272dd P 2b98ac09d43149c5bcfd2c0080ea5379 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 07:58:41.941322 4457 raft_consensus.cc:1203] T fa6cf4447cfe49ecbbfe94accb24aa97 P 98526b95c3e94668a1673fe660495876 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 07:58:42.015908 3911 raft_consensus.cc:1203] T 102891a28f95444dac4798b2aace3857 P 9846630212094f49930ef629ee885368 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 07:58:42.065301 3911 raft_consensus.cc:1203] T ef28ebd14a684f1f9f5d6718bf8284b4 P 9846630212094f49930ef629ee885368 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 07:58:42.154712 4054 raft_consensus.cc:1203] T 0b2690f2189b43f9a45e1a90605623ad P cb0a6f265d7e4362aea53c916ceb49ec [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 07:58:42.199333 4326 raft_consensus.cc:1203] T c64dae3606e049e2a72b00148fffc748 P 2471afee7ac94728a1506dc907ef8fbb [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 07:58:42.217528 4054 raft_consensus.cc:1203] T 149f1f36b61a4d148c2003caaa7fce0c P cb0a6f265d7e4362aea53c916ceb49ec [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 07:58:42.229475 4457 raft_consensus.cc:1203] T db9747650ebf4556ac30e4138821695b P 98526b95c3e94668a1673fe660495876 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 07:58:42.230273 4326 raft_consensus.cc:1203] T c63214bc45b3424492cc114662f6839e P 2471afee7ac94728a1506dc907ef8fbb [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 07:58:42.373270 4644 raft_consensus.cc:1050] T fa6cf4447cfe49ecbbfe94accb24aa97 P 809d92b226a94aa59287ddc81c7ff77b: attempting to promote NON_VOTER 98526b95c3e94668a1673fe660495876 to VOTER
I20240501 07:58:42.373869 4644 consensus_queue.cc:237] T fa6cf4447cfe49ecbbfe94accb24aa97 P 809d92b226a94aa59287ddc81c7ff77b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "809d92b226a94aa59287ddc81c7ff77b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33451 } attrs { replace: true } } peers { permanent_uuid: "98526b95c3e94668a1673fe660495876" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39073 } attrs { promote: false } }
I20240501 07:58:42.375180 4457 raft_consensus.cc:1261] T fa6cf4447cfe49ecbbfe94accb24aa97 P 98526b95c3e94668a1673fe660495876 [term 1 LEARNER]: Refusing update from remote peer 809d92b226a94aa59287ddc81c7ff77b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:58:42.375699 4854 consensus_queue.cc:1035] T fa6cf4447cfe49ecbbfe94accb24aa97 P 809d92b226a94aa59287ddc81c7ff77b [LEADER]: Connected to new peer: Peer: permanent_uuid: "98526b95c3e94668a1673fe660495876" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39073 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:58:42.378482 4644 raft_consensus.cc:2932] T fa6cf4447cfe49ecbbfe94accb24aa97 P 809d92b226a94aa59287ddc81c7ff77b [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 98526b95c3e94668a1673fe660495876 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "809d92b226a94aa59287ddc81c7ff77b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33451 } attrs { replace: true } } peers { permanent_uuid: "98526b95c3e94668a1673fe660495876" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39073 } attrs { promote: false } } }
I20240501 07:58:42.378991 4457 raft_consensus.cc:2932] T fa6cf4447cfe49ecbbfe94accb24aa97 P 98526b95c3e94668a1673fe660495876 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 98526b95c3e94668a1673fe660495876 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "809d92b226a94aa59287ddc81c7ff77b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33451 } attrs { replace: true } } peers { permanent_uuid: "98526b95c3e94668a1673fe660495876" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39073 } attrs { promote: false } } }
I20240501 07:58:42.381369 2641 catalog_manager.cc:5484] T fa6cf4447cfe49ecbbfe94accb24aa97 P 98526b95c3e94668a1673fe660495876 reported cstate change: config changed from index 4 to 5, 98526b95c3e94668a1673fe660495876 (127.0.99.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "809d92b226a94aa59287ddc81c7ff77b" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "809d92b226a94aa59287ddc81c7ff77b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33451 } attrs { replace: true } } peers { permanent_uuid: "98526b95c3e94668a1673fe660495876" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39073 } attrs { promote: false } } }
I20240501 07:58:42.416179 4854 raft_consensus.cc:1050] T 18f0be6c9b944ce1b63600f456f272dd P 809d92b226a94aa59287ddc81c7ff77b: attempting to promote NON_VOTER 2b98ac09d43149c5bcfd2c0080ea5379 to VOTER
I20240501 07:58:42.416579 4854 consensus_queue.cc:237] T 18f0be6c9b944ce1b63600f456f272dd P 809d92b226a94aa59287ddc81c7ff77b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "809d92b226a94aa59287ddc81c7ff77b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33451 } attrs { replace: true } } peers { permanent_uuid: "2b98ac09d43149c5bcfd2c0080ea5379" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 40367 } attrs { promote: false } }
I20240501 07:58:42.417810 4186 raft_consensus.cc:1261] T 18f0be6c9b944ce1b63600f456f272dd P 2b98ac09d43149c5bcfd2c0080ea5379 [term 1 LEARNER]: Refusing update from remote peer 809d92b226a94aa59287ddc81c7ff77b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:58:42.418213 4853 consensus_queue.cc:1035] T 18f0be6c9b944ce1b63600f456f272dd P 809d92b226a94aa59287ddc81c7ff77b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b98ac09d43149c5bcfd2c0080ea5379" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 40367 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:58:42.420063 4644 raft_consensus.cc:2932] T 18f0be6c9b944ce1b63600f456f272dd P 809d92b226a94aa59287ddc81c7ff77b [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2b98ac09d43149c5bcfd2c0080ea5379 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "809d92b226a94aa59287ddc81c7ff77b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33451 } attrs { replace: true } } peers { permanent_uuid: "2b98ac09d43149c5bcfd2c0080ea5379" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 40367 } attrs { promote: false } } }
I20240501 07:58:42.421659 4186 raft_consensus.cc:2932] T 18f0be6c9b944ce1b63600f456f272dd P 2b98ac09d43149c5bcfd2c0080ea5379 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2b98ac09d43149c5bcfd2c0080ea5379 (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "809d92b226a94aa59287ddc81c7ff77b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33451 } attrs { replace: true } } peers { permanent_uuid: "2b98ac09d43149c5bcfd2c0080ea5379" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 40367 } attrs { promote: false } } }
I20240501 07:58:42.422322 2649 catalog_manager.cc:5484] T 18f0be6c9b944ce1b63600f456f272dd P 809d92b226a94aa59287ddc81c7ff77b reported cstate change: config changed from index 4 to 5, 2b98ac09d43149c5bcfd2c0080ea5379 (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "809d92b226a94aa59287ddc81c7ff77b" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "809d92b226a94aa59287ddc81c7ff77b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33451 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2b98ac09d43149c5bcfd2c0080ea5379" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 40367 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 07:58:42.433265 2791 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "fa6cf4447cfe49ecbbfe94accb24aa97"
dest_uuid: "809d92b226a94aa59287ddc81c7ff77b"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:59400
I20240501 07:58:42.433444 2791 raft_consensus.cc:595] T fa6cf4447cfe49ecbbfe94accb24aa97 P 809d92b226a94aa59287ddc81c7ff77b [term 1 LEADER]: Received request to transfer leadership
I20240501 07:58:42.443531 2791 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "18f0be6c9b944ce1b63600f456f272dd"
dest_uuid: "809d92b226a94aa59287ddc81c7ff77b"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:59412
I20240501 07:58:42.443673 2791 raft_consensus.cc:595] T 18f0be6c9b944ce1b63600f456f272dd P 809d92b226a94aa59287ddc81c7ff77b [term 1 LEADER]: Received request to transfer leadership
I20240501 07:58:42.514124 4593 raft_consensus.cc:1050] T 102891a28f95444dac4798b2aace3857 P 853826bd6abf488a8895392f3470a546: attempting to promote NON_VOTER 9846630212094f49930ef629ee885368 to VOTER
I20240501 07:58:42.514669 4593 consensus_queue.cc:237] T 102891a28f95444dac4798b2aace3857 P 853826bd6abf488a8895392f3470a546 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "853826bd6abf488a8895392f3470a546" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39801 } attrs { replace: true } } peers { permanent_uuid: "9846630212094f49930ef629ee885368" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33467 } attrs { promote: false } }
I20240501 07:58:42.516016 3911 raft_consensus.cc:1261] T 102891a28f95444dac4798b2aace3857 P 9846630212094f49930ef629ee885368 [term 1 LEARNER]: Refusing update from remote peer 853826bd6abf488a8895392f3470a546: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:58:42.516507 4900 consensus_queue.cc:1035] T 102891a28f95444dac4798b2aace3857 P 853826bd6abf488a8895392f3470a546 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9846630212094f49930ef629ee885368" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33467 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:58:42.519202 4593 raft_consensus.cc:2932] T 102891a28f95444dac4798b2aace3857 P 853826bd6abf488a8895392f3470a546 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9846630212094f49930ef629ee885368 (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "853826bd6abf488a8895392f3470a546" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39801 } attrs { replace: true } } peers { permanent_uuid: "9846630212094f49930ef629ee885368" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33467 } attrs { promote: false } } }
I20240501 07:58:42.519801 3911 raft_consensus.cc:2932] T 102891a28f95444dac4798b2aace3857 P 9846630212094f49930ef629ee885368 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9846630212094f49930ef629ee885368 (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "853826bd6abf488a8895392f3470a546" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39801 } attrs { replace: true } } peers { permanent_uuid: "9846630212094f49930ef629ee885368" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33467 } attrs { promote: false } } }
I20240501 07:58:42.521778 2644 catalog_manager.cc:5484] T 102891a28f95444dac4798b2aace3857 P 853826bd6abf488a8895392f3470a546 reported cstate change: config changed from index 4 to 5, 9846630212094f49930ef629ee885368 (127.0.99.131) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "853826bd6abf488a8895392f3470a546" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "853826bd6abf488a8895392f3470a546" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39801 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9846630212094f49930ef629ee885368" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33467 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 07:58:42.556488 4593 raft_consensus.cc:1050] T ef28ebd14a684f1f9f5d6718bf8284b4 P 853826bd6abf488a8895392f3470a546: attempting to promote NON_VOTER 9846630212094f49930ef629ee885368 to VOTER
I20240501 07:58:42.556824 4593 consensus_queue.cc:237] T ef28ebd14a684f1f9f5d6718bf8284b4 P 853826bd6abf488a8895392f3470a546 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "853826bd6abf488a8895392f3470a546" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39801 } attrs { replace: true } } peers { permanent_uuid: "9846630212094f49930ef629ee885368" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33467 } attrs { promote: false } }
I20240501 07:58:42.557960 3911 raft_consensus.cc:1261] T ef28ebd14a684f1f9f5d6718bf8284b4 P 9846630212094f49930ef629ee885368 [term 1 LEARNER]: Refusing update from remote peer 853826bd6abf488a8895392f3470a546: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:58:42.558346 4593 consensus_queue.cc:1035] T ef28ebd14a684f1f9f5d6718bf8284b4 P 853826bd6abf488a8895392f3470a546 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9846630212094f49930ef629ee885368" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33467 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:58:42.560037 4901 raft_consensus.cc:2932] T ef28ebd14a684f1f9f5d6718bf8284b4 P 853826bd6abf488a8895392f3470a546 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9846630212094f49930ef629ee885368 (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "853826bd6abf488a8895392f3470a546" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39801 } attrs { replace: true } } peers { permanent_uuid: "9846630212094f49930ef629ee885368" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33467 } attrs { promote: false } } }
I20240501 07:58:42.560559 3911 raft_consensus.cc:2932] T ef28ebd14a684f1f9f5d6718bf8284b4 P 9846630212094f49930ef629ee885368 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9846630212094f49930ef629ee885368 (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "853826bd6abf488a8895392f3470a546" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39801 } attrs { replace: true } } peers { permanent_uuid: "9846630212094f49930ef629ee885368" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33467 } attrs { promote: false } } }
I20240501 07:58:42.562811 2649 catalog_manager.cc:5484] T ef28ebd14a684f1f9f5d6718bf8284b4 P 853826bd6abf488a8895392f3470a546 reported cstate change: config changed from index 4 to 5, 9846630212094f49930ef629ee885368 (127.0.99.131) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "853826bd6abf488a8895392f3470a546" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "853826bd6abf488a8895392f3470a546" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39801 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9846630212094f49930ef629ee885368" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33467 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 07:58:42.661080 2922 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ef28ebd14a684f1f9f5d6718bf8284b4"
dest_uuid: "853826bd6abf488a8895392f3470a546"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:42724
I20240501 07:58:42.661252 2922 raft_consensus.cc:595] T ef28ebd14a684f1f9f5d6718bf8284b4 P 853826bd6abf488a8895392f3470a546 [term 1 LEADER]: Received request to transfer leadership
I20240501 07:58:42.695083 4900 raft_consensus.cc:1050] T 0b2690f2189b43f9a45e1a90605623ad P 853826bd6abf488a8895392f3470a546: attempting to promote NON_VOTER cb0a6f265d7e4362aea53c916ceb49ec to VOTER
I20240501 07:58:42.695432 4900 consensus_queue.cc:237] T 0b2690f2189b43f9a45e1a90605623ad P 853826bd6abf488a8895392f3470a546 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "853826bd6abf488a8895392f3470a546" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39801 } attrs { replace: true } } peers { permanent_uuid: "cb0a6f265d7e4362aea53c916ceb49ec" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43375 } attrs { promote: false } }
I20240501 07:58:42.696285 2922 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "102891a28f95444dac4798b2aace3857"
dest_uuid: "853826bd6abf488a8895392f3470a546"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:42756
I20240501 07:58:42.696403 2922 raft_consensus.cc:595] T 102891a28f95444dac4798b2aace3857 P 853826bd6abf488a8895392f3470a546 [term 1 LEADER]: Received request to transfer leadership
I20240501 07:58:42.696655 4054 raft_consensus.cc:1261] T 0b2690f2189b43f9a45e1a90605623ad P cb0a6f265d7e4362aea53c916ceb49ec [term 1 LEARNER]: Refusing update from remote peer 853826bd6abf488a8895392f3470a546: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:58:42.697254 4900 consensus_queue.cc:1035] T 0b2690f2189b43f9a45e1a90605623ad P 853826bd6abf488a8895392f3470a546 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb0a6f265d7e4362aea53c916ceb49ec" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43375 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:58:42.700182 4593 raft_consensus.cc:2932] T 0b2690f2189b43f9a45e1a90605623ad P 853826bd6abf488a8895392f3470a546 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, cb0a6f265d7e4362aea53c916ceb49ec (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "853826bd6abf488a8895392f3470a546" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39801 } attrs { replace: true } } peers { permanent_uuid: "cb0a6f265d7e4362aea53c916ceb49ec" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43375 } attrs { promote: false } } }
I20240501 07:58:42.701572 4054 raft_consensus.cc:2932] T 0b2690f2189b43f9a45e1a90605623ad P cb0a6f265d7e4362aea53c916ceb49ec [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, cb0a6f265d7e4362aea53c916ceb49ec (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "853826bd6abf488a8895392f3470a546" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39801 } attrs { replace: true } } peers { permanent_uuid: "cb0a6f265d7e4362aea53c916ceb49ec" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43375 } attrs { promote: false } } }
I20240501 07:58:42.704056 2644 catalog_manager.cc:5484] T 0b2690f2189b43f9a45e1a90605623ad P cb0a6f265d7e4362aea53c916ceb49ec reported cstate change: config changed from index 4 to 5, cb0a6f265d7e4362aea53c916ceb49ec (127.0.99.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "853826bd6abf488a8895392f3470a546" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "853826bd6abf488a8895392f3470a546" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39801 } attrs { replace: true } } peers { permanent_uuid: "cb0a6f265d7e4362aea53c916ceb49ec" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43375 } attrs { promote: false } } }
I20240501 07:58:42.709602 2791 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "fa6cf4447cfe49ecbbfe94accb24aa97"
dest_uuid: "809d92b226a94aa59287ddc81c7ff77b"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:59444
I20240501 07:58:42.709733 2791 raft_consensus.cc:595] T fa6cf4447cfe49ecbbfe94accb24aa97 P 809d92b226a94aa59287ddc81c7ff77b [term 1 LEADER]: Received request to transfer leadership
I20240501 07:58:42.719980 2791 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "18f0be6c9b944ce1b63600f456f272dd"
dest_uuid: "809d92b226a94aa59287ddc81c7ff77b"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:59458
I20240501 07:58:42.720119 2791 raft_consensus.cc:595] T 18f0be6c9b944ce1b63600f456f272dd P 809d92b226a94aa59287ddc81c7ff77b [term 1 LEADER]: Received request to transfer leadership
I20240501 07:58:42.746454 4853 raft_consensus.cc:1050] T c64dae3606e049e2a72b00148fffc748 P 809d92b226a94aa59287ddc81c7ff77b: attempting to promote NON_VOTER 2471afee7ac94728a1506dc907ef8fbb to VOTER
I20240501 07:58:42.746824 4853 consensus_queue.cc:237] T c64dae3606e049e2a72b00148fffc748 P 809d92b226a94aa59287ddc81c7ff77b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "809d92b226a94aa59287ddc81c7ff77b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33451 } attrs { replace: true } } peers { permanent_uuid: "2471afee7ac94728a1506dc907ef8fbb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38167 } attrs { promote: false } }
I20240501 07:58:42.747964 4326 raft_consensus.cc:1261] T c64dae3606e049e2a72b00148fffc748 P 2471afee7ac94728a1506dc907ef8fbb [term 1 LEARNER]: Refusing update from remote peer 809d92b226a94aa59287ddc81c7ff77b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:58:42.748394 4853 consensus_queue.cc:1035] T c64dae3606e049e2a72b00148fffc748 P 809d92b226a94aa59287ddc81c7ff77b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2471afee7ac94728a1506dc907ef8fbb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38167 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:58:42.751086 4854 raft_consensus.cc:2932] T c64dae3606e049e2a72b00148fffc748 P 809d92b226a94aa59287ddc81c7ff77b [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2471afee7ac94728a1506dc907ef8fbb (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "809d92b226a94aa59287ddc81c7ff77b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33451 } attrs { replace: true } } peers { permanent_uuid: "2471afee7ac94728a1506dc907ef8fbb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38167 } attrs { promote: false } } }
I20240501 07:58:42.751648 4326 raft_consensus.cc:2932] T c64dae3606e049e2a72b00148fffc748 P 2471afee7ac94728a1506dc907ef8fbb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2471afee7ac94728a1506dc907ef8fbb (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "809d92b226a94aa59287ddc81c7ff77b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33451 } attrs { replace: true } } peers { permanent_uuid: "2471afee7ac94728a1506dc907ef8fbb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38167 } attrs { promote: false } } }
I20240501 07:58:42.753718 2649 catalog_manager.cc:5484] T c64dae3606e049e2a72b00148fffc748 P 809d92b226a94aa59287ddc81c7ff77b reported cstate change: config changed from index 4 to 5, 2471afee7ac94728a1506dc907ef8fbb (127.0.99.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "809d92b226a94aa59287ddc81c7ff77b" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "809d92b226a94aa59287ddc81c7ff77b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33451 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2471afee7ac94728a1506dc907ef8fbb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38167 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 07:58:42.760254 4644 raft_consensus.cc:1050] T 149f1f36b61a4d148c2003caaa7fce0c P 809d92b226a94aa59287ddc81c7ff77b: attempting to promote NON_VOTER cb0a6f265d7e4362aea53c916ceb49ec to VOTER
I20240501 07:58:42.760618 4644 consensus_queue.cc:237] T 149f1f36b61a4d148c2003caaa7fce0c P 809d92b226a94aa59287ddc81c7ff77b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "809d92b226a94aa59287ddc81c7ff77b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33451 } attrs { replace: true } } peers { permanent_uuid: "cb0a6f265d7e4362aea53c916ceb49ec" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43375 } attrs { promote: false } }
I20240501 07:58:42.761732 4054 raft_consensus.cc:1261] T 149f1f36b61a4d148c2003caaa7fce0c P cb0a6f265d7e4362aea53c916ceb49ec [term 1 LEARNER]: Refusing update from remote peer 809d92b226a94aa59287ddc81c7ff77b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:58:42.762092 4644 consensus_queue.cc:1035] T 149f1f36b61a4d148c2003caaa7fce0c P 809d92b226a94aa59287ddc81c7ff77b [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb0a6f265d7e4362aea53c916ceb49ec" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43375 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:58:42.763825 4854 raft_consensus.cc:2932] T 149f1f36b61a4d148c2003caaa7fce0c P 809d92b226a94aa59287ddc81c7ff77b [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, cb0a6f265d7e4362aea53c916ceb49ec (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "809d92b226a94aa59287ddc81c7ff77b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33451 } attrs { replace: true } } peers { permanent_uuid: "cb0a6f265d7e4362aea53c916ceb49ec" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43375 } attrs { promote: false } } }
I20240501 07:58:42.765486 4853 raft_consensus.cc:1050] T db9747650ebf4556ac30e4138821695b P 809d92b226a94aa59287ddc81c7ff77b: attempting to promote NON_VOTER 98526b95c3e94668a1673fe660495876 to VOTER
I20240501 07:58:42.765655 4054 raft_consensus.cc:2932] T 149f1f36b61a4d148c2003caaa7fce0c P cb0a6f265d7e4362aea53c916ceb49ec [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, cb0a6f265d7e4362aea53c916ceb49ec (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "809d92b226a94aa59287ddc81c7ff77b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33451 } attrs { replace: true } } peers { permanent_uuid: "cb0a6f265d7e4362aea53c916ceb49ec" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43375 } attrs { promote: false } } }
I20240501 07:58:42.765781 4853 consensus_queue.cc:237] T db9747650ebf4556ac30e4138821695b P 809d92b226a94aa59287ddc81c7ff77b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "809d92b226a94aa59287ddc81c7ff77b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33451 } attrs { replace: true } } peers { permanent_uuid: "98526b95c3e94668a1673fe660495876" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39073 } attrs { promote: false } }
I20240501 07:58:42.766757 2644 catalog_manager.cc:5484] T 149f1f36b61a4d148c2003caaa7fce0c P 809d92b226a94aa59287ddc81c7ff77b reported cstate change: config changed from index 4 to 5, cb0a6f265d7e4362aea53c916ceb49ec (127.0.99.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "809d92b226a94aa59287ddc81c7ff77b" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "809d92b226a94aa59287ddc81c7ff77b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33451 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cb0a6f265d7e4362aea53c916ceb49ec" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43375 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 07:58:42.767112 4457 raft_consensus.cc:1261] T db9747650ebf4556ac30e4138821695b P 98526b95c3e94668a1673fe660495876 [term 1 LEARNER]: Refusing update from remote peer 809d92b226a94aa59287ddc81c7ff77b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:58:42.768163 4644 consensus_queue.cc:1035] T db9747650ebf4556ac30e4138821695b P 809d92b226a94aa59287ddc81c7ff77b [LEADER]: Connected to new peer: Peer: permanent_uuid: "98526b95c3e94668a1673fe660495876" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39073 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:58:42.770390 4853 raft_consensus.cc:2932] T db9747650ebf4556ac30e4138821695b P 809d92b226a94aa59287ddc81c7ff77b [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 98526b95c3e94668a1673fe660495876 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "809d92b226a94aa59287ddc81c7ff77b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33451 } attrs { replace: true } } peers { permanent_uuid: "98526b95c3e94668a1673fe660495876" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39073 } attrs { promote: false } } }
I20240501 07:58:42.770972 4457 raft_consensus.cc:2932] T db9747650ebf4556ac30e4138821695b P 98526b95c3e94668a1673fe660495876 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 98526b95c3e94668a1673fe660495876 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "809d92b226a94aa59287ddc81c7ff77b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33451 } attrs { replace: true } } peers { permanent_uuid: "98526b95c3e94668a1673fe660495876" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39073 } attrs { promote: false } } }
I20240501 07:58:42.773279 2649 catalog_manager.cc:5484] T db9747650ebf4556ac30e4138821695b P 98526b95c3e94668a1673fe660495876 reported cstate change: config changed from index 4 to 5, 98526b95c3e94668a1673fe660495876 (127.0.99.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "809d92b226a94aa59287ddc81c7ff77b" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "809d92b226a94aa59287ddc81c7ff77b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33451 } attrs { replace: true } } peers { permanent_uuid: "98526b95c3e94668a1673fe660495876" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39073 } attrs { promote: false } } }
I20240501 07:58:42.806492 4644 raft_consensus.cc:979] T 18f0be6c9b944ce1b63600f456f272dd P 809d92b226a94aa59287ddc81c7ff77b: : Instructing follower 2b98ac09d43149c5bcfd2c0080ea5379 to start an election
I20240501 07:58:42.806653 4644 raft_consensus.cc:1067] T 18f0be6c9b944ce1b63600f456f272dd P 809d92b226a94aa59287ddc81c7ff77b [term 1 LEADER]: Signalling peer 2b98ac09d43149c5bcfd2c0080ea5379 to start an election
I20240501 07:58:42.807216 4186 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "18f0be6c9b944ce1b63600f456f272dd"
dest_uuid: "2b98ac09d43149c5bcfd2c0080ea5379"
from {username='slave'} at 127.0.99.129:49077
I20240501 07:58:42.807401 4186 raft_consensus.cc:482] T 18f0be6c9b944ce1b63600f456f272dd P 2b98ac09d43149c5bcfd2c0080ea5379 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:58:42.807475 4186 raft_consensus.cc:3037] T 18f0be6c9b944ce1b63600f456f272dd P 2b98ac09d43149c5bcfd2c0080ea5379 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:58:42.808303 4186 raft_consensus.cc:504] T 18f0be6c9b944ce1b63600f456f272dd P 2b98ac09d43149c5bcfd2c0080ea5379 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "809d92b226a94aa59287ddc81c7ff77b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33451 } attrs { replace: true } } peers { permanent_uuid: "2b98ac09d43149c5bcfd2c0080ea5379" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 40367 } attrs { promote: false } }
I20240501 07:58:42.809046 4186 leader_election.cc:288] T 18f0be6c9b944ce1b63600f456f272dd P 2b98ac09d43149c5bcfd2c0080ea5379 [CANDIDATE]: Term 2 election: Requested vote from peers 809d92b226a94aa59287ddc81c7ff77b (127.0.99.129:33451)
I20240501 07:58:42.809485 2791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18f0be6c9b944ce1b63600f456f272dd" candidate_uuid: "2b98ac09d43149c5bcfd2c0080ea5379" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "809d92b226a94aa59287ddc81c7ff77b"
I20240501 07:58:42.809650 2791 raft_consensus.cc:3032] T 18f0be6c9b944ce1b63600f456f272dd P 809d92b226a94aa59287ddc81c7ff77b [term 1 LEADER]: Stepping down as leader of term 1
I20240501 07:58:42.809695 2791 raft_consensus.cc:729] T 18f0be6c9b944ce1b63600f456f272dd P 809d92b226a94aa59287ddc81c7ff77b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 809d92b226a94aa59287ddc81c7ff77b, State: Running, Role: LEADER
I20240501 07:58:42.809856 2791 consensus_queue.cc:260] T 18f0be6c9b944ce1b63600f456f272dd P 809d92b226a94aa59287ddc81c7ff77b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "809d92b226a94aa59287ddc81c7ff77b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33451 } attrs { replace: true } } peers { permanent_uuid: "2b98ac09d43149c5bcfd2c0080ea5379" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 40367 } attrs { promote: false } }
I20240501 07:58:42.810003 2791 raft_consensus.cc:3037] T 18f0be6c9b944ce1b63600f456f272dd P 809d92b226a94aa59287ddc81c7ff77b [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:58:42.810847 2791 raft_consensus.cc:2454] T 18f0be6c9b944ce1b63600f456f272dd P 809d92b226a94aa59287ddc81c7ff77b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b98ac09d43149c5bcfd2c0080ea5379 in term 2.
I20240501 07:58:42.811215 4121 leader_election.cc:302] T 18f0be6c9b944ce1b63600f456f272dd P 2b98ac09d43149c5bcfd2c0080ea5379 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 2b98ac09d43149c5bcfd2c0080ea5379, 809d92b226a94aa59287ddc81c7ff77b; no voters:
I20240501 07:58:42.811417 4985 raft_consensus.cc:2781] T 18f0be6c9b944ce1b63600f456f272dd P 2b98ac09d43149c5bcfd2c0080ea5379 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 07:58:42.811627 4985 raft_consensus.cc:686] T 18f0be6c9b944ce1b63600f456f272dd P 2b98ac09d43149c5bcfd2c0080ea5379 [term 2 LEADER]: Becoming Leader. State: Replica: 2b98ac09d43149c5bcfd2c0080ea5379, State: Running, Role: LEADER
I20240501 07:58:42.811779 4985 consensus_queue.cc:237] T 18f0be6c9b944ce1b63600f456f272dd P 2b98ac09d43149c5bcfd2c0080ea5379 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "809d92b226a94aa59287ddc81c7ff77b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33451 } attrs { replace: true } } peers { permanent_uuid: "2b98ac09d43149c5bcfd2c0080ea5379" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 40367 } attrs { promote: false } }
I20240501 07:58:42.813496 2644 catalog_manager.cc:5484] T 18f0be6c9b944ce1b63600f456f272dd P 2b98ac09d43149c5bcfd2c0080ea5379 reported cstate change: term changed from 1 to 2, leader changed from 809d92b226a94aa59287ddc81c7ff77b (127.0.99.129) to 2b98ac09d43149c5bcfd2c0080ea5379 (127.0.99.133). New cstate: current_term: 2 leader_uuid: "2b98ac09d43149c5bcfd2c0080ea5379" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "809d92b226a94aa59287ddc81c7ff77b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33451 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2b98ac09d43149c5bcfd2c0080ea5379" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 40367 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240501 07:58:42.820667 2635 catalog_manager.cc:5014] ChangeConfig:REMOVE_PEER RPC for tablet 18f0be6c9b944ce1b63600f456f272dd with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER failed with leader 2b98ac09d43149c5bcfd2c0080ea5379 (127.0.99.133:40367) due to error UNKNOWN_ERROR; will retry: Illegal state: Leader has not yet committed an operation in its own term
I20240501 07:58:42.823824 4593 raft_consensus.cc:1050] T c63214bc45b3424492cc114662f6839e P 853826bd6abf488a8895392f3470a546: attempting to promote NON_VOTER 2471afee7ac94728a1506dc907ef8fbb to VOTER
I20240501 07:58:42.824167 4593 consensus_queue.cc:237] T c63214bc45b3424492cc114662f6839e P 853826bd6abf488a8895392f3470a546 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "853826bd6abf488a8895392f3470a546" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39801 } attrs { replace: true } } peers { permanent_uuid: "2471afee7ac94728a1506dc907ef8fbb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38167 } attrs { promote: false } }
I20240501 07:58:42.825241 4326 raft_consensus.cc:1261] T c63214bc45b3424492cc114662f6839e P 2471afee7ac94728a1506dc907ef8fbb [term 1 LEARNER]: Refusing update from remote peer 853826bd6abf488a8895392f3470a546: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:58:42.825650 4593 consensus_queue.cc:1035] T c63214bc45b3424492cc114662f6839e P 853826bd6abf488a8895392f3470a546 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2471afee7ac94728a1506dc907ef8fbb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38167 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:58:42.827347 4901 raft_consensus.cc:2932] T c63214bc45b3424492cc114662f6839e P 853826bd6abf488a8895392f3470a546 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2471afee7ac94728a1506dc907ef8fbb (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "853826bd6abf488a8895392f3470a546" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39801 } attrs { replace: true } } peers { permanent_uuid: "2471afee7ac94728a1506dc907ef8fbb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38167 } attrs { promote: false } } }
I20240501 07:58:42.827769 4326 raft_consensus.cc:2932] T c63214bc45b3424492cc114662f6839e P 2471afee7ac94728a1506dc907ef8fbb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2471afee7ac94728a1506dc907ef8fbb (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "853826bd6abf488a8895392f3470a546" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39801 } attrs { replace: true } } peers { permanent_uuid: "2471afee7ac94728a1506dc907ef8fbb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38167 } attrs { promote: false } } }
I20240501 07:58:42.829690 2644 catalog_manager.cc:5484] T c63214bc45b3424492cc114662f6839e P 853826bd6abf488a8895392f3470a546 reported cstate change: config changed from index 4 to 5, 2471afee7ac94728a1506dc907ef8fbb (127.0.99.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "853826bd6abf488a8895392f3470a546" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "853826bd6abf488a8895392f3470a546" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39801 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2471afee7ac94728a1506dc907ef8fbb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38167 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 07:58:42.937024 2922 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ef28ebd14a684f1f9f5d6718bf8284b4"
dest_uuid: "853826bd6abf488a8895392f3470a546"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:42772
I20240501 07:58:42.937182 2922 raft_consensus.cc:595] T ef28ebd14a684f1f9f5d6718bf8284b4 P 853826bd6abf488a8895392f3470a546 [term 1 LEADER]: Received request to transfer leadership
I20240501 07:58:42.947396 2922 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c63214bc45b3424492cc114662f6839e"
dest_uuid: "853826bd6abf488a8895392f3470a546"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:42788
I20240501 07:58:42.947595 2922 raft_consensus.cc:595] T c63214bc45b3424492cc114662f6839e P 853826bd6abf488a8895392f3470a546 [term 1 LEADER]: Received request to transfer leadership
I20240501 07:58:42.958843 2791 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c64dae3606e049e2a72b00148fffc748"
dest_uuid: "809d92b226a94aa59287ddc81c7ff77b"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:59472
I20240501 07:58:42.958988 2791 raft_consensus.cc:595] T c64dae3606e049e2a72b00148fffc748 P 809d92b226a94aa59287ddc81c7ff77b [term 1 LEADER]: Received request to transfer leadership
I20240501 07:58:42.963070 4901 raft_consensus.cc:979] T ef28ebd14a684f1f9f5d6718bf8284b4 P 853826bd6abf488a8895392f3470a546: : Instructing follower 9846630212094f49930ef629ee885368 to start an election
I20240501 07:58:42.963189 4901 raft_consensus.cc:1067] T ef28ebd14a684f1f9f5d6718bf8284b4 P 853826bd6abf488a8895392f3470a546 [term 1 LEADER]: Signalling peer 9846630212094f49930ef629ee885368 to start an election
I20240501 07:58:42.963824 3911 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ef28ebd14a684f1f9f5d6718bf8284b4"
dest_uuid: "9846630212094f49930ef629ee885368"
from {username='slave'} at 127.0.99.130:60657
I20240501 07:58:42.964035 3911 raft_consensus.cc:482] T ef28ebd14a684f1f9f5d6718bf8284b4 P 9846630212094f49930ef629ee885368 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:58:42.964175 3911 raft_consensus.cc:3037] T ef28ebd14a684f1f9f5d6718bf8284b4 P 9846630212094f49930ef629ee885368 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:58:42.965234 3911 raft_consensus.cc:504] T ef28ebd14a684f1f9f5d6718bf8284b4 P 9846630212094f49930ef629ee885368 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "853826bd6abf488a8895392f3470a546" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39801 } attrs { replace: true } } peers { permanent_uuid: "9846630212094f49930ef629ee885368" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33467 } attrs { promote: false } }
I20240501 07:58:42.966300 3911 leader_election.cc:288] T ef28ebd14a684f1f9f5d6718bf8284b4 P 9846630212094f49930ef629ee885368 [CANDIDATE]: Term 2 election: Requested vote from peers 853826bd6abf488a8895392f3470a546 (127.0.99.130:39801)
I20240501 07:58:42.966678 2922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef28ebd14a684f1f9f5d6718bf8284b4" candidate_uuid: "9846630212094f49930ef629ee885368" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "853826bd6abf488a8895392f3470a546"
I20240501 07:58:42.966835 2922 raft_consensus.cc:3032] T ef28ebd14a684f1f9f5d6718bf8284b4 P 853826bd6abf488a8895392f3470a546 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 07:58:42.966887 2922 raft_consensus.cc:729] T ef28ebd14a684f1f9f5d6718bf8284b4 P 853826bd6abf488a8895392f3470a546 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 853826bd6abf488a8895392f3470a546, State: Running, Role: LEADER
I20240501 07:58:42.967056 2922 consensus_queue.cc:260] T ef28ebd14a684f1f9f5d6718bf8284b4 P 853826bd6abf488a8895392f3470a546 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "853826bd6abf488a8895392f3470a546" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39801 } attrs { replace: true } } peers { permanent_uuid: "9846630212094f49930ef629ee885368" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33467 } attrs { promote: false } }
I20240501 07:58:42.967211 2922 raft_consensus.cc:3037] T ef28ebd14a684f1f9f5d6718bf8284b4 P 853826bd6abf488a8895392f3470a546 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:58:42.968056 2922 raft_consensus.cc:2454] T ef28ebd14a684f1f9f5d6718bf8284b4 P 853826bd6abf488a8895392f3470a546 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9846630212094f49930ef629ee885368 in term 2.
I20240501 07:58:42.968446 3859 leader_election.cc:302] T ef28ebd14a684f1f9f5d6718bf8284b4 P 9846630212094f49930ef629ee885368 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 853826bd6abf488a8895392f3470a546, 9846630212094f49930ef629ee885368; no voters:
I20240501 07:58:42.968681 4771 raft_consensus.cc:2781] T ef28ebd14a684f1f9f5d6718bf8284b4 P 9846630212094f49930ef629ee885368 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 07:58:42.968858 4771 raft_consensus.cc:686] T ef28ebd14a684f1f9f5d6718bf8284b4 P 9846630212094f49930ef629ee885368 [term 2 LEADER]: Becoming Leader. State: Replica: 9846630212094f49930ef629ee885368, State: Running, Role: LEADER
I20240501 07:58:42.969173 4771 consensus_queue.cc:237] T ef28ebd14a684f1f9f5d6718bf8284b4 P 9846630212094f49930ef629ee885368 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "853826bd6abf488a8895392f3470a546" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39801 } attrs { replace: true } } peers { permanent_uuid: "9846630212094f49930ef629ee885368" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33467 } attrs { promote: false } }
I20240501 07:58:42.971061 2649 catalog_manager.cc:5484] T ef28ebd14a684f1f9f5d6718bf8284b4 P 9846630212094f49930ef629ee885368 reported cstate change: term changed from 1 to 2, leader changed from 853826bd6abf488a8895392f3470a546 (127.0.99.130) to 9846630212094f49930ef629ee885368 (127.0.99.131). New cstate: current_term: 2 leader_uuid: "9846630212094f49930ef629ee885368" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "853826bd6abf488a8895392f3470a546" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39801 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9846630212094f49930ef629ee885368" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33467 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 07:58:42.972936 2922 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0b2690f2189b43f9a45e1a90605623ad"
dest_uuid: "853826bd6abf488a8895392f3470a546"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:42800
I20240501 07:58:42.973120 2922 raft_consensus.cc:595] T 0b2690f2189b43f9a45e1a90605623ad P 853826bd6abf488a8895392f3470a546 [term 1 LEADER]: Received request to transfer leadership
I20240501 07:58:42.983777 2791 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "db9747650ebf4556ac30e4138821695b"
dest_uuid: "809d92b226a94aa59287ddc81c7ff77b"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:59490
I20240501 07:58:42.983943 2791 raft_consensus.cc:595] T db9747650ebf4556ac30e4138821695b P 809d92b226a94aa59287ddc81c7ff77b [term 1 LEADER]: Received request to transfer leadership
I20240501 07:58:42.991748 4853 raft_consensus.cc:979] T fa6cf4447cfe49ecbbfe94accb24aa97 P 809d92b226a94aa59287ddc81c7ff77b: : Instructing follower 98526b95c3e94668a1673fe660495876 to start an election
I20240501 07:58:42.991894 4644 raft_consensus.cc:1067] T fa6cf4447cfe49ecbbfe94accb24aa97 P 809d92b226a94aa59287ddc81c7ff77b [term 1 LEADER]: Signalling peer 98526b95c3e94668a1673fe660495876 to start an election
I20240501 07:58:42.992381 4457 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "fa6cf4447cfe49ecbbfe94accb24aa97"
dest_uuid: "98526b95c3e94668a1673fe660495876"
from {username='slave'} at 127.0.99.129:60159
I20240501 07:58:42.992594 4457 raft_consensus.cc:482] T fa6cf4447cfe49ecbbfe94accb24aa97 P 98526b95c3e94668a1673fe660495876 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:58:42.992718 4457 raft_consensus.cc:3037] T fa6cf4447cfe49ecbbfe94accb24aa97 P 98526b95c3e94668a1673fe660495876 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:58:42.993814 4457 raft_consensus.cc:504] T fa6cf4447cfe49ecbbfe94accb24aa97 P 98526b95c3e94668a1673fe660495876 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "809d92b226a94aa59287ddc81c7ff77b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33451 } attrs { replace: true } } peers { permanent_uuid: "98526b95c3e94668a1673fe660495876" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39073 } attrs { promote: false } }
I20240501 07:58:42.995178 4457 leader_election.cc:288] T fa6cf4447cfe49ecbbfe94accb24aa97 P 98526b95c3e94668a1673fe660495876 [CANDIDATE]: Term 2 election: Requested vote from peers 809d92b226a94aa59287ddc81c7ff77b (127.0.99.129:33451)
I20240501 07:58:42.995421 2922 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "102891a28f95444dac4798b2aace3857"
dest_uuid: "853826bd6abf488a8895392f3470a546"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:42828
I20240501 07:58:42.995561 2922 raft_consensus.cc:595] T 102891a28f95444dac4798b2aace3857 P 853826bd6abf488a8895392f3470a546 [term 1 LEADER]: Received request to transfer leadership
I20240501 07:58:42.995553 2791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa6cf4447cfe49ecbbfe94accb24aa97" candidate_uuid: "98526b95c3e94668a1673fe660495876" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "809d92b226a94aa59287ddc81c7ff77b"
I20240501 07:58:42.995702 2791 raft_consensus.cc:3032] T fa6cf4447cfe49ecbbfe94accb24aa97 P 809d92b226a94aa59287ddc81c7ff77b [term 1 LEADER]: Stepping down as leader of term 1
I20240501 07:58:42.995760 2791 raft_consensus.cc:729] T fa6cf4447cfe49ecbbfe94accb24aa97 P 809d92b226a94aa59287ddc81c7ff77b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 809d92b226a94aa59287ddc81c7ff77b, State: Running, Role: LEADER
I20240501 07:58:42.995946 2791 consensus_queue.cc:260] T fa6cf4447cfe49ecbbfe94accb24aa97 P 809d92b226a94aa59287ddc81c7ff77b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "809d92b226a94aa59287ddc81c7ff77b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33451 } attrs { replace: true } } peers { permanent_uuid: "98526b95c3e94668a1673fe660495876" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39073 } attrs { promote: false } }
I20240501 07:58:42.996129 2791 raft_consensus.cc:3037] T fa6cf4447cfe49ecbbfe94accb24aa97 P 809d92b226a94aa59287ddc81c7ff77b [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:58:42.997321 2791 raft_consensus.cc:2454] T fa6cf4447cfe49ecbbfe94accb24aa97 P 809d92b226a94aa59287ddc81c7ff77b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98526b95c3e94668a1673fe660495876 in term 2.
I20240501 07:58:42.997681 4393 leader_election.cc:302] T fa6cf4447cfe49ecbbfe94accb24aa97 P 98526b95c3e94668a1673fe660495876 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 809d92b226a94aa59287ddc81c7ff77b, 98526b95c3e94668a1673fe660495876; no voters:
I20240501 07:58:42.997928 4754 raft_consensus.cc:2781] T fa6cf4447cfe49ecbbfe94accb24aa97 P 98526b95c3e94668a1673fe660495876 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 07:58:42.998108 4754 raft_consensus.cc:686] T fa6cf4447cfe49ecbbfe94accb24aa97 P 98526b95c3e94668a1673fe660495876 [term 2 LEADER]: Becoming Leader. State: Replica: 98526b95c3e94668a1673fe660495876, State: Running, Role: LEADER
I20240501 07:58:42.998450 4754 consensus_queue.cc:237] T fa6cf4447cfe49ecbbfe94accb24aa97 P 98526b95c3e94668a1673fe660495876 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "809d92b226a94aa59287ddc81c7ff77b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33451 } attrs { replace: true } } peers { permanent_uuid: "98526b95c3e94668a1673fe660495876" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39073 } attrs { promote: false } }
I20240501 07:58:43.000378 2649 catalog_manager.cc:5484] T fa6cf4447cfe49ecbbfe94accb24aa97 P 98526b95c3e94668a1673fe660495876 reported cstate change: term changed from 1 to 2, leader changed from 809d92b226a94aa59287ddc81c7ff77b (127.0.99.129) to 98526b95c3e94668a1673fe660495876 (127.0.99.135). New cstate: current_term: 2 leader_uuid: "98526b95c3e94668a1673fe660495876" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "809d92b226a94aa59287ddc81c7ff77b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33451 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "98526b95c3e94668a1673fe660495876" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39073 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 07:58:43.022514 2791 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "149f1f36b61a4d148c2003caaa7fce0c"
dest_uuid: "809d92b226a94aa59287ddc81c7ff77b"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:59524
I20240501 07:58:43.022698 2791 raft_consensus.cc:595] T 149f1f36b61a4d148c2003caaa7fce0c P 809d92b226a94aa59287ddc81c7ff77b [term 1 LEADER]: Received request to transfer leadership
I20240501 07:58:43.069993 4900 raft_consensus.cc:979] T 102891a28f95444dac4798b2aace3857 P 853826bd6abf488a8895392f3470a546: : Instructing follower 9846630212094f49930ef629ee885368 to start an election
I20240501 07:58:43.070108 4900 raft_consensus.cc:1067] T 102891a28f95444dac4798b2aace3857 P 853826bd6abf488a8895392f3470a546 [term 1 LEADER]: Signalling peer 9846630212094f49930ef629ee885368 to start an election
I20240501 07:58:43.070561 3911 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "102891a28f95444dac4798b2aace3857"
dest_uuid: "9846630212094f49930ef629ee885368"
from {username='slave'} at 127.0.99.130:60657
I20240501 07:58:43.070719 3911 raft_consensus.cc:482] T 102891a28f95444dac4798b2aace3857 P 9846630212094f49930ef629ee885368 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:58:43.070768 3911 raft_consensus.cc:3037] T 102891a28f95444dac4798b2aace3857 P 9846630212094f49930ef629ee885368 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:58:43.071671 3911 raft_consensus.cc:504] T 102891a28f95444dac4798b2aace3857 P 9846630212094f49930ef629ee885368 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "853826bd6abf488a8895392f3470a546" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39801 } attrs { replace: true } } peers { permanent_uuid: "9846630212094f49930ef629ee885368" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33467 } attrs { promote: false } }
I20240501 07:58:43.071956 3911 leader_election.cc:288] T 102891a28f95444dac4798b2aace3857 P 9846630212094f49930ef629ee885368 [CANDIDATE]: Term 2 election: Requested vote from peers 853826bd6abf488a8895392f3470a546 (127.0.99.130:39801)
I20240501 07:58:43.072357 2922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "102891a28f95444dac4798b2aace3857" candidate_uuid: "9846630212094f49930ef629ee885368" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "853826bd6abf488a8895392f3470a546"
I20240501 07:58:43.072480 2922 raft_consensus.cc:3032] T 102891a28f95444dac4798b2aace3857 P 853826bd6abf488a8895392f3470a546 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 07:58:43.072530 2922 raft_consensus.cc:729] T 102891a28f95444dac4798b2aace3857 P 853826bd6abf488a8895392f3470a546 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 853826bd6abf488a8895392f3470a546, State: Running, Role: LEADER
I20240501 07:58:43.072679 2922 consensus_queue.cc:260] T 102891a28f95444dac4798b2aace3857 P 853826bd6abf488a8895392f3470a546 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "853826bd6abf488a8895392f3470a546" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39801 } attrs { replace: true } } peers { permanent_uuid: "9846630212094f49930ef629ee885368" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33467 } attrs { promote: false } }
I20240501 07:58:43.072814 2922 raft_consensus.cc:3037] T 102891a28f95444dac4798b2aace3857 P 853826bd6abf488a8895392f3470a546 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:58:43.073666 2922 raft_consensus.cc:2454] T 102891a28f95444dac4798b2aace3857 P 853826bd6abf488a8895392f3470a546 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9846630212094f49930ef629ee885368 in term 2.
I20240501 07:58:43.074023 3859 leader_election.cc:302] T 102891a28f95444dac4798b2aace3857 P 9846630212094f49930ef629ee885368 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 853826bd6abf488a8895392f3470a546, 9846630212094f49930ef629ee885368; no voters:
I20240501 07:58:43.074235 4771 raft_consensus.cc:2781] T 102891a28f95444dac4798b2aace3857 P 9846630212094f49930ef629ee885368 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 07:58:43.074323 4771 raft_consensus.cc:686] T 102891a28f95444dac4798b2aace3857 P 9846630212094f49930ef629ee885368 [term 2 LEADER]: Becoming Leader. State: Replica: 9846630212094f49930ef629ee885368, State: Running, Role: LEADER
I20240501 07:58:43.074465 4771 consensus_queue.cc:237] T 102891a28f95444dac4798b2aace3857 P 9846630212094f49930ef629ee885368 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "853826bd6abf488a8895392f3470a546" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39801 } attrs { replace: true } } peers { permanent_uuid: "9846630212094f49930ef629ee885368" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33467 } attrs { promote: false } }
I20240501 07:58:43.075891 2649 catalog_manager.cc:5484] T 102891a28f95444dac4798b2aace3857 P 9846630212094f49930ef629ee885368 reported cstate change: term changed from 1 to 2, leader changed from 853826bd6abf488a8895392f3470a546 (127.0.99.130) to 9846630212094f49930ef629ee885368 (127.0.99.131). New cstate: current_term: 2 leader_uuid: "9846630212094f49930ef629ee885368" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "853826bd6abf488a8895392f3470a546" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39801 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9846630212094f49930ef629ee885368" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33467 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 07:58:43.098506 4901 raft_consensus.cc:979] T 0b2690f2189b43f9a45e1a90605623ad P 853826bd6abf488a8895392f3470a546: : Instructing follower cb0a6f265d7e4362aea53c916ceb49ec to start an election
I20240501 07:58:43.098640 4901 raft_consensus.cc:1067] T 0b2690f2189b43f9a45e1a90605623ad P 853826bd6abf488a8895392f3470a546 [term 1 LEADER]: Signalling peer cb0a6f265d7e4362aea53c916ceb49ec to start an election
I20240501 07:58:43.099115 4054 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0b2690f2189b43f9a45e1a90605623ad"
dest_uuid: "cb0a6f265d7e4362aea53c916ceb49ec"
from {username='slave'} at 127.0.99.130:35217
I20240501 07:58:43.099315 4054 raft_consensus.cc:482] T 0b2690f2189b43f9a45e1a90605623ad P cb0a6f265d7e4362aea53c916ceb49ec [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:58:43.099417 4054 raft_consensus.cc:3037] T 0b2690f2189b43f9a45e1a90605623ad P cb0a6f265d7e4362aea53c916ceb49ec [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:58:43.100487 4054 raft_consensus.cc:504] T 0b2690f2189b43f9a45e1a90605623ad P cb0a6f265d7e4362aea53c916ceb49ec [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "853826bd6abf488a8895392f3470a546" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39801 } attrs { replace: true } } peers { permanent_uuid: "cb0a6f265d7e4362aea53c916ceb49ec" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43375 } attrs { promote: false } }
I20240501 07:58:43.101522 4054 leader_election.cc:288] T 0b2690f2189b43f9a45e1a90605623ad P cb0a6f265d7e4362aea53c916ceb49ec [CANDIDATE]: Term 2 election: Requested vote from peers 853826bd6abf488a8895392f3470a546 (127.0.99.130:39801)
I20240501 07:58:43.102003 2922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b2690f2189b43f9a45e1a90605623ad" candidate_uuid: "cb0a6f265d7e4362aea53c916ceb49ec" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "853826bd6abf488a8895392f3470a546"
I20240501 07:58:43.102133 2922 raft_consensus.cc:3032] T 0b2690f2189b43f9a45e1a90605623ad P 853826bd6abf488a8895392f3470a546 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 07:58:43.102185 2922 raft_consensus.cc:729] T 0b2690f2189b43f9a45e1a90605623ad P 853826bd6abf488a8895392f3470a546 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 853826bd6abf488a8895392f3470a546, State: Running, Role: LEADER
I20240501 07:58:43.102339 2922 consensus_queue.cc:260] T 0b2690f2189b43f9a45e1a90605623ad P 853826bd6abf488a8895392f3470a546 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "853826bd6abf488a8895392f3470a546" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39801 } attrs { replace: true } } peers { permanent_uuid: "cb0a6f265d7e4362aea53c916ceb49ec" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43375 } attrs { promote: false } }
I20240501 07:58:43.102474 2922 raft_consensus.cc:3037] T 0b2690f2189b43f9a45e1a90605623ad P 853826bd6abf488a8895392f3470a546 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:58:43.103283 2922 raft_consensus.cc:2454] T 0b2690f2189b43f9a45e1a90605623ad P 853826bd6abf488a8895392f3470a546 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb0a6f265d7e4362aea53c916ceb49ec in term 2.
I20240501 07:58:43.103660 3990 leader_election.cc:302] T 0b2690f2189b43f9a45e1a90605623ad P cb0a6f265d7e4362aea53c916ceb49ec [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 853826bd6abf488a8895392f3470a546, cb0a6f265d7e4362aea53c916ceb49ec; no voters:
I20240501 07:58:43.103914 4829 raft_consensus.cc:2781] T 0b2690f2189b43f9a45e1a90605623ad P cb0a6f265d7e4362aea53c916ceb49ec [term 2 FOLLOWER]: Leader election won for term 2
I20240501 07:58:43.104118 4829 raft_consensus.cc:686] T 0b2690f2189b43f9a45e1a90605623ad P cb0a6f265d7e4362aea53c916ceb49ec [term 2 LEADER]: Becoming Leader. State: Replica: cb0a6f265d7e4362aea53c916ceb49ec, State: Running, Role: LEADER
I20240501 07:58:43.104424 4829 consensus_queue.cc:237] T 0b2690f2189b43f9a45e1a90605623ad P cb0a6f265d7e4362aea53c916ceb49ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "853826bd6abf488a8895392f3470a546" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39801 } attrs { replace: true } } peers { permanent_uuid: "cb0a6f265d7e4362aea53c916ceb49ec" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43375 } attrs { promote: false } }
I20240501 07:58:43.106874 2649 catalog_manager.cc:5484] T 0b2690f2189b43f9a45e1a90605623ad P cb0a6f265d7e4362aea53c916ceb49ec reported cstate change: term changed from 1 to 2, leader changed from 853826bd6abf488a8895392f3470a546 (127.0.99.130) to cb0a6f265d7e4362aea53c916ceb49ec (127.0.99.132). New cstate: current_term: 2 leader_uuid: "cb0a6f265d7e4362aea53c916ceb49ec" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "853826bd6abf488a8895392f3470a546" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39801 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cb0a6f265d7e4362aea53c916ceb49ec" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43375 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 07:58:43.211352 2791 raft_consensus.cc:1261] T 18f0be6c9b944ce1b63600f456f272dd P 809d92b226a94aa59287ddc81c7ff77b [term 2 FOLLOWER]: Refusing update from remote peer 2b98ac09d43149c5bcfd2c0080ea5379: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 07:58:43.212018 4985 consensus_queue.cc:1035] T 18f0be6c9b944ce1b63600f456f272dd P 2b98ac09d43149c5bcfd2c0080ea5379 [LEADER]: Connected to new peer: Peer: permanent_uuid: "809d92b226a94aa59287ddc81c7ff77b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33451 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 07:58:43.220425 4853 raft_consensus.cc:979] T 149f1f36b61a4d148c2003caaa7fce0c P 809d92b226a94aa59287ddc81c7ff77b: : Instructing follower cb0a6f265d7e4362aea53c916ceb49ec to start an election
I20240501 07:58:43.220547 4853 raft_consensus.cc:1067] T 149f1f36b61a4d148c2003caaa7fce0c P 809d92b226a94aa59287ddc81c7ff77b [term 1 LEADER]: Signalling peer cb0a6f265d7e4362aea53c916ceb49ec to start an election
I20240501 07:58:43.221004 4054 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "149f1f36b61a4d148c2003caaa7fce0c"
dest_uuid: "cb0a6f265d7e4362aea53c916ceb49ec"
from {username='slave'} at 127.0.99.129:45515
I20240501 07:58:43.221144 4054 raft_consensus.cc:482] T 149f1f36b61a4d148c2003caaa7fce0c P cb0a6f265d7e4362aea53c916ceb49ec [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:58:43.221213 4054 raft_consensus.cc:3037] T 149f1f36b61a4d148c2003caaa7fce0c P cb0a6f265d7e4362aea53c916ceb49ec [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:58:43.222050 4054 raft_consensus.cc:504] T 149f1f36b61a4d148c2003caaa7fce0c P cb0a6f265d7e4362aea53c916ceb49ec [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "809d92b226a94aa59287ddc81c7ff77b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33451 } attrs { replace: true } } peers { permanent_uuid: "cb0a6f265d7e4362aea53c916ceb49ec" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43375 } attrs { promote: false } }
I20240501 07:58:43.222410 4054 leader_election.cc:288] T 149f1f36b61a4d148c2003caaa7fce0c P cb0a6f265d7e4362aea53c916ceb49ec [CANDIDATE]: Term 2 election: Requested vote from peers 809d92b226a94aa59287ddc81c7ff77b (127.0.99.129:33451)
I20240501 07:58:43.222872 2791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "149f1f36b61a4d148c2003caaa7fce0c" candidate_uuid: "cb0a6f265d7e4362aea53c916ceb49ec" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "809d92b226a94aa59287ddc81c7ff77b"
I20240501 07:58:43.222985 2791 raft_consensus.cc:3032] T 149f1f36b61a4d148c2003caaa7fce0c P 809d92b226a94aa59287ddc81c7ff77b [term 1 LEADER]: Stepping down as leader of term 1
I20240501 07:58:43.223035 2791 raft_consensus.cc:729] T 149f1f36b61a4d148c2003caaa7fce0c P 809d92b226a94aa59287ddc81c7ff77b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 809d92b226a94aa59287ddc81c7ff77b, State: Running, Role: LEADER
I20240501 07:58:43.223176 2791 consensus_queue.cc:260] T 149f1f36b61a4d148c2003caaa7fce0c P 809d92b226a94aa59287ddc81c7ff77b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "809d92b226a94aa59287ddc81c7ff77b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33451 } attrs { replace: true } } peers { permanent_uuid: "cb0a6f265d7e4362aea53c916ceb49ec" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43375 } attrs { promote: false } }
I20240501 07:58:43.223302 2791 raft_consensus.cc:3037] T 149f1f36b61a4d148c2003caaa7fce0c P 809d92b226a94aa59287ddc81c7ff77b [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:58:43.224275 2791 raft_consensus.cc:2454] T 149f1f36b61a4d148c2003caaa7fce0c P 809d92b226a94aa59287ddc81c7ff77b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb0a6f265d7e4362aea53c916ceb49ec in term 2.
I20240501 07:58:43.224568 3990 leader_election.cc:302] T 149f1f36b61a4d148c2003caaa7fce0c P cb0a6f265d7e4362aea53c916ceb49ec [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 809d92b226a94aa59287ddc81c7ff77b, cb0a6f265d7e4362aea53c916ceb49ec; no voters:
I20240501 07:58:43.224730 4829 raft_consensus.cc:2781] T 149f1f36b61a4d148c2003caaa7fce0c P cb0a6f265d7e4362aea53c916ceb49ec [term 2 FOLLOWER]: Leader election won for term 2
I20240501 07:58:43.224838 4829 raft_consensus.cc:686] T 149f1f36b61a4d148c2003caaa7fce0c P cb0a6f265d7e4362aea53c916ceb49ec [term 2 LEADER]: Becoming Leader. State: Replica: cb0a6f265d7e4362aea53c916ceb49ec, State: Running, Role: LEADER
I20240501 07:58:43.225016 4829 consensus_queue.cc:237] T 149f1f36b61a4d148c2003caaa7fce0c P cb0a6f265d7e4362aea53c916ceb49ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "809d92b226a94aa59287ddc81c7ff77b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33451 } attrs { replace: true } } peers { permanent_uuid: "cb0a6f265d7e4362aea53c916ceb49ec" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43375 } attrs { promote: false } }
I20240501 07:58:43.226897 2644 catalog_manager.cc:5484] T 149f1f36b61a4d148c2003caaa7fce0c P cb0a6f265d7e4362aea53c916ceb49ec reported cstate change: term changed from 1 to 2, leader changed from 809d92b226a94aa59287ddc81c7ff77b (127.0.99.129) to cb0a6f265d7e4362aea53c916ceb49ec (127.0.99.132). New cstate: current_term: 2 leader_uuid: "cb0a6f265d7e4362aea53c916ceb49ec" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "809d92b226a94aa59287ddc81c7ff77b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33451 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cb0a6f265d7e4362aea53c916ceb49ec" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43375 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 07:58:43.240427 2922 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c63214bc45b3424492cc114662f6839e"
dest_uuid: "853826bd6abf488a8895392f3470a546"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:42846
I20240501 07:58:43.240576 2922 raft_consensus.cc:595] T c63214bc45b3424492cc114662f6839e P 853826bd6abf488a8895392f3470a546 [term 1 LEADER]: Received request to transfer leadership
I20240501 07:58:43.250746 2791 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c64dae3606e049e2a72b00148fffc748"
dest_uuid: "809d92b226a94aa59287ddc81c7ff77b"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:59540
I20240501 07:58:43.250887 2791 raft_consensus.cc:595] T c64dae3606e049e2a72b00148fffc748 P 809d92b226a94aa59287ddc81c7ff77b [term 1 LEADER]: Received request to transfer leadership
I20240501 07:58:43.266988 2791 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "db9747650ebf4556ac30e4138821695b"
dest_uuid: "809d92b226a94aa59287ddc81c7ff77b"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:59550
I20240501 07:58:43.267146 2791 raft_consensus.cc:595] T db9747650ebf4556ac30e4138821695b P 809d92b226a94aa59287ddc81c7ff77b [term 1 LEADER]: Received request to transfer leadership
I20240501 07:58:43.297609 4644 raft_consensus.cc:979] T c64dae3606e049e2a72b00148fffc748 P 809d92b226a94aa59287ddc81c7ff77b: : Instructing follower 2471afee7ac94728a1506dc907ef8fbb to start an election
I20240501 07:58:43.297741 4644 raft_consensus.cc:1067] T c64dae3606e049e2a72b00148fffc748 P 809d92b226a94aa59287ddc81c7ff77b [term 1 LEADER]: Signalling peer 2471afee7ac94728a1506dc907ef8fbb to start an election
I20240501 07:58:43.298223 4326 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c64dae3606e049e2a72b00148fffc748"
dest_uuid: "2471afee7ac94728a1506dc907ef8fbb"
from {username='slave'} at 127.0.99.129:45585
I20240501 07:58:43.298399 4326 raft_consensus.cc:482] T c64dae3606e049e2a72b00148fffc748 P 2471afee7ac94728a1506dc907ef8fbb [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:58:43.298483 4326 raft_consensus.cc:3037] T c64dae3606e049e2a72b00148fffc748 P 2471afee7ac94728a1506dc907ef8fbb [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:58:43.299281 4326 raft_consensus.cc:504] T c64dae3606e049e2a72b00148fffc748 P 2471afee7ac94728a1506dc907ef8fbb [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "809d92b226a94aa59287ddc81c7ff77b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33451 } attrs { replace: true } } peers { permanent_uuid: "2471afee7ac94728a1506dc907ef8fbb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38167 } attrs { promote: false } }
I20240501 07:58:43.300151 4326 leader_election.cc:288] T c64dae3606e049e2a72b00148fffc748 P 2471afee7ac94728a1506dc907ef8fbb [CANDIDATE]: Term 2 election: Requested vote from peers 809d92b226a94aa59287ddc81c7ff77b (127.0.99.129:33451)
I20240501 07:58:43.300529 2791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c64dae3606e049e2a72b00148fffc748" candidate_uuid: "2471afee7ac94728a1506dc907ef8fbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "809d92b226a94aa59287ddc81c7ff77b"
I20240501 07:58:43.300645 2791 raft_consensus.cc:3032] T c64dae3606e049e2a72b00148fffc748 P 809d92b226a94aa59287ddc81c7ff77b [term 1 LEADER]: Stepping down as leader of term 1
I20240501 07:58:43.300675 2791 raft_consensus.cc:729] T c64dae3606e049e2a72b00148fffc748 P 809d92b226a94aa59287ddc81c7ff77b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 809d92b226a94aa59287ddc81c7ff77b, State: Running, Role: LEADER
I20240501 07:58:43.300823 2791 consensus_queue.cc:260] T c64dae3606e049e2a72b00148fffc748 P 809d92b226a94aa59287ddc81c7ff77b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "809d92b226a94aa59287ddc81c7ff77b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33451 } attrs { replace: true } } peers { permanent_uuid: "2471afee7ac94728a1506dc907ef8fbb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38167 } attrs { promote: false } }
I20240501 07:58:43.300948 2791 raft_consensus.cc:3037] T c64dae3606e049e2a72b00148fffc748 P 809d92b226a94aa59287ddc81c7ff77b [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:58:43.301740 2791 raft_consensus.cc:2454] T c64dae3606e049e2a72b00148fffc748 P 809d92b226a94aa59287ddc81c7ff77b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2471afee7ac94728a1506dc907ef8fbb in term 2.
I20240501 07:58:43.302102 4262 leader_election.cc:302] T c64dae3606e049e2a72b00148fffc748 P 2471afee7ac94728a1506dc907ef8fbb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 2471afee7ac94728a1506dc907ef8fbb, 809d92b226a94aa59287ddc81c7ff77b; no voters:
I20240501 07:58:43.302350 5084 raft_consensus.cc:2781] T c64dae3606e049e2a72b00148fffc748 P 2471afee7ac94728a1506dc907ef8fbb [term 2 FOLLOWER]: Leader election won for term 2
I20240501 07:58:43.302529 5084 raft_consensus.cc:686] T c64dae3606e049e2a72b00148fffc748 P 2471afee7ac94728a1506dc907ef8fbb [term 2 LEADER]: Becoming Leader. State: Replica: 2471afee7ac94728a1506dc907ef8fbb, State: Running, Role: LEADER
I20240501 07:58:43.302814 5084 consensus_queue.cc:237] T c64dae3606e049e2a72b00148fffc748 P 2471afee7ac94728a1506dc907ef8fbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "809d92b226a94aa59287ddc81c7ff77b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33451 } attrs { replace: true } } peers { permanent_uuid: "2471afee7ac94728a1506dc907ef8fbb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38167 } attrs { promote: false } }
I20240501 07:58:43.305099 2644 catalog_manager.cc:5484] T c64dae3606e049e2a72b00148fffc748 P 2471afee7ac94728a1506dc907ef8fbb reported cstate change: term changed from 1 to 2, leader changed from 809d92b226a94aa59287ddc81c7ff77b (127.0.99.129) to 2471afee7ac94728a1506dc907ef8fbb (127.0.99.134). New cstate: current_term: 2 leader_uuid: "2471afee7ac94728a1506dc907ef8fbb" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "809d92b226a94aa59287ddc81c7ff77b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33451 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2471afee7ac94728a1506dc907ef8fbb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38167 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 07:58:43.320533 4853 raft_consensus.cc:979] T db9747650ebf4556ac30e4138821695b P 809d92b226a94aa59287ddc81c7ff77b: : Instructing follower 98526b95c3e94668a1673fe660495876 to start an election
I20240501 07:58:43.320641 4853 raft_consensus.cc:1067] T db9747650ebf4556ac30e4138821695b P 809d92b226a94aa59287ddc81c7ff77b [term 1 LEADER]: Signalling peer 98526b95c3e94668a1673fe660495876 to start an election
I20240501 07:58:43.321085 4457 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "db9747650ebf4556ac30e4138821695b"
dest_uuid: "98526b95c3e94668a1673fe660495876"
from {username='slave'} at 127.0.99.129:60159
I20240501 07:58:43.321210 4457 raft_consensus.cc:482] T db9747650ebf4556ac30e4138821695b P 98526b95c3e94668a1673fe660495876 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:58:43.321256 4457 raft_consensus.cc:3037] T db9747650ebf4556ac30e4138821695b P 98526b95c3e94668a1673fe660495876 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:58:43.322044 4457 raft_consensus.cc:504] T db9747650ebf4556ac30e4138821695b P 98526b95c3e94668a1673fe660495876 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "809d92b226a94aa59287ddc81c7ff77b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33451 } attrs { replace: true } } peers { permanent_uuid: "98526b95c3e94668a1673fe660495876" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39073 } attrs { promote: false } }
I20240501 07:58:43.322310 4457 leader_election.cc:288] T db9747650ebf4556ac30e4138821695b P 98526b95c3e94668a1673fe660495876 [CANDIDATE]: Term 2 election: Requested vote from peers 809d92b226a94aa59287ddc81c7ff77b (127.0.99.129:33451)
I20240501 07:58:43.322679 2791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db9747650ebf4556ac30e4138821695b" candidate_uuid: "98526b95c3e94668a1673fe660495876" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "809d92b226a94aa59287ddc81c7ff77b"
I20240501 07:58:43.322800 2791 raft_consensus.cc:3032] T db9747650ebf4556ac30e4138821695b P 809d92b226a94aa59287ddc81c7ff77b [term 1 LEADER]: Stepping down as leader of term 1
I20240501 07:58:43.322846 2791 raft_consensus.cc:729] T db9747650ebf4556ac30e4138821695b P 809d92b226a94aa59287ddc81c7ff77b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 809d92b226a94aa59287ddc81c7ff77b, State: Running, Role: LEADER
I20240501 07:58:43.322957 2791 consensus_queue.cc:260] T db9747650ebf4556ac30e4138821695b P 809d92b226a94aa59287ddc81c7ff77b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "809d92b226a94aa59287ddc81c7ff77b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33451 } attrs { replace: true } } peers { permanent_uuid: "98526b95c3e94668a1673fe660495876" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39073 } attrs { promote: false } }
I20240501 07:58:43.323064 2791 raft_consensus.cc:3037] T db9747650ebf4556ac30e4138821695b P 809d92b226a94aa59287ddc81c7ff77b [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:58:43.323771 2791 raft_consensus.cc:2454] T db9747650ebf4556ac30e4138821695b P 809d92b226a94aa59287ddc81c7ff77b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98526b95c3e94668a1673fe660495876 in term 2.
I20240501 07:58:43.324059 4393 leader_election.cc:302] T db9747650ebf4556ac30e4138821695b P 98526b95c3e94668a1673fe660495876 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 809d92b226a94aa59287ddc81c7ff77b, 98526b95c3e94668a1673fe660495876; no voters:
I20240501 07:58:43.324223 4754 raft_consensus.cc:2781] T db9747650ebf4556ac30e4138821695b P 98526b95c3e94668a1673fe660495876 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 07:58:43.324333 4754 raft_consensus.cc:686] T db9747650ebf4556ac30e4138821695b P 98526b95c3e94668a1673fe660495876 [term 2 LEADER]: Becoming Leader. State: Replica: 98526b95c3e94668a1673fe660495876, State: Running, Role: LEADER
I20240501 07:58:43.324478 4754 consensus_queue.cc:237] T db9747650ebf4556ac30e4138821695b P 98526b95c3e94668a1673fe660495876 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "809d92b226a94aa59287ddc81c7ff77b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33451 } attrs { replace: true } } peers { permanent_uuid: "98526b95c3e94668a1673fe660495876" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39073 } attrs { promote: false } }
I20240501 07:58:43.325970 2644 catalog_manager.cc:5484] T db9747650ebf4556ac30e4138821695b P 98526b95c3e94668a1673fe660495876 reported cstate change: term changed from 1 to 2, leader changed from 809d92b226a94aa59287ddc81c7ff77b (127.0.99.129) to 98526b95c3e94668a1673fe660495876 (127.0.99.135). New cstate: current_term: 2 leader_uuid: "98526b95c3e94668a1673fe660495876" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "809d92b226a94aa59287ddc81c7ff77b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33451 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "98526b95c3e94668a1673fe660495876" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39073 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 07:58:43.400480 2791 raft_consensus.cc:1261] T fa6cf4447cfe49ecbbfe94accb24aa97 P 809d92b226a94aa59287ddc81c7ff77b [term 2 FOLLOWER]: Refusing update from remote peer 98526b95c3e94668a1673fe660495876: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 07:58:43.401047 4754 consensus_queue.cc:1035] T fa6cf4447cfe49ecbbfe94accb24aa97 P 98526b95c3e94668a1673fe660495876 [LEADER]: Connected to new peer: Peer: permanent_uuid: "809d92b226a94aa59287ddc81c7ff77b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33451 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 07:58:43.419437 4900 raft_consensus.cc:979] T c63214bc45b3424492cc114662f6839e P 853826bd6abf488a8895392f3470a546: : Instructing follower 2471afee7ac94728a1506dc907ef8fbb to start an election
I20240501 07:58:43.419545 4900 raft_consensus.cc:1067] T c63214bc45b3424492cc114662f6839e P 853826bd6abf488a8895392f3470a546 [term 1 LEADER]: Signalling peer 2471afee7ac94728a1506dc907ef8fbb to start an election
I20240501 07:58:43.420013 4326 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c63214bc45b3424492cc114662f6839e"
dest_uuid: "2471afee7ac94728a1506dc907ef8fbb"
from {username='slave'} at 127.0.99.130:45695
I20240501 07:58:43.420147 4326 raft_consensus.cc:482] T c63214bc45b3424492cc114662f6839e P 2471afee7ac94728a1506dc907ef8fbb [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:58:43.420195 4326 raft_consensus.cc:3037] T c63214bc45b3424492cc114662f6839e P 2471afee7ac94728a1506dc907ef8fbb [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:58:43.421030 4326 raft_consensus.cc:504] T c63214bc45b3424492cc114662f6839e P 2471afee7ac94728a1506dc907ef8fbb [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "853826bd6abf488a8895392f3470a546" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39801 } attrs { replace: true } } peers { permanent_uuid: "2471afee7ac94728a1506dc907ef8fbb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38167 } attrs { promote: false } }
I20240501 07:58:43.421372 4326 leader_election.cc:288] T c63214bc45b3424492cc114662f6839e P 2471afee7ac94728a1506dc907ef8fbb [CANDIDATE]: Term 2 election: Requested vote from peers 853826bd6abf488a8895392f3470a546 (127.0.99.130:39801)
I20240501 07:58:43.421769 2922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c63214bc45b3424492cc114662f6839e" candidate_uuid: "2471afee7ac94728a1506dc907ef8fbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "853826bd6abf488a8895392f3470a546"
I20240501 07:58:43.421898 2922 raft_consensus.cc:3032] T c63214bc45b3424492cc114662f6839e P 853826bd6abf488a8895392f3470a546 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 07:58:43.421950 2922 raft_consensus.cc:729] T c63214bc45b3424492cc114662f6839e P 853826bd6abf488a8895392f3470a546 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 853826bd6abf488a8895392f3470a546, State: Running, Role: LEADER
I20240501 07:58:43.422103 2922 consensus_queue.cc:260] T c63214bc45b3424492cc114662f6839e P 853826bd6abf488a8895392f3470a546 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "853826bd6abf488a8895392f3470a546" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39801 } attrs { replace: true } } peers { permanent_uuid: "2471afee7ac94728a1506dc907ef8fbb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38167 } attrs { promote: false } }
I20240501 07:58:43.422237 2922 raft_consensus.cc:3037] T c63214bc45b3424492cc114662f6839e P 853826bd6abf488a8895392f3470a546 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:58:43.422994 2922 raft_consensus.cc:2454] T c63214bc45b3424492cc114662f6839e P 853826bd6abf488a8895392f3470a546 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2471afee7ac94728a1506dc907ef8fbb in term 2.
I20240501 07:58:43.423297 4262 leader_election.cc:302] T c63214bc45b3424492cc114662f6839e P 2471afee7ac94728a1506dc907ef8fbb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 2471afee7ac94728a1506dc907ef8fbb, 853826bd6abf488a8895392f3470a546; no voters:
I20240501 07:58:43.423478 5084 raft_consensus.cc:2781] T c63214bc45b3424492cc114662f6839e P 2471afee7ac94728a1506dc907ef8fbb [term 2 FOLLOWER]: Leader election won for term 2
I20240501 07:58:43.423565 5084 raft_consensus.cc:686] T c63214bc45b3424492cc114662f6839e P 2471afee7ac94728a1506dc907ef8fbb [term 2 LEADER]: Becoming Leader. State: Replica: 2471afee7ac94728a1506dc907ef8fbb, State: Running, Role: LEADER
I20240501 07:58:43.423708 5084 consensus_queue.cc:237] T c63214bc45b3424492cc114662f6839e P 2471afee7ac94728a1506dc907ef8fbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "853826bd6abf488a8895392f3470a546" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39801 } attrs { replace: true } } peers { permanent_uuid: "2471afee7ac94728a1506dc907ef8fbb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38167 } attrs { promote: false } }
I20240501 07:58:43.425132 2649 catalog_manager.cc:5484] T c63214bc45b3424492cc114662f6839e P 2471afee7ac94728a1506dc907ef8fbb reported cstate change: term changed from 1 to 2, leader changed from 853826bd6abf488a8895392f3470a546 (127.0.99.130) to 2471afee7ac94728a1506dc907ef8fbb (127.0.99.134). New cstate: current_term: 2 leader_uuid: "2471afee7ac94728a1506dc907ef8fbb" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "853826bd6abf488a8895392f3470a546" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39801 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2471afee7ac94728a1506dc907ef8fbb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38167 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 07:58:43.465771 4186 consensus_queue.cc:237] T 18f0be6c9b944ce1b63600f456f272dd P 2b98ac09d43149c5bcfd2c0080ea5379 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2b98ac09d43149c5bcfd2c0080ea5379" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 40367 } attrs { promote: false } }
I20240501 07:58:43.466491 5089 raft_consensus.cc:2932] T 18f0be6c9b944ce1b63600f456f272dd P 2b98ac09d43149c5bcfd2c0080ea5379 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 809d92b226a94aa59287ddc81c7ff77b (127.0.99.129) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2b98ac09d43149c5bcfd2c0080ea5379" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 40367 } attrs { promote: false } } }
I20240501 07:58:43.467803 2635 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 18f0be6c9b944ce1b63600f456f272dd with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240501 07:58:43.468770 2649 catalog_manager.cc:5484] T 18f0be6c9b944ce1b63600f456f272dd P 2b98ac09d43149c5bcfd2c0080ea5379 reported cstate change: config changed from index 5 to 7, VOTER 809d92b226a94aa59287ddc81c7ff77b (127.0.99.129) evicted. New cstate: current_term: 2 leader_uuid: "2b98ac09d43149c5bcfd2c0080ea5379" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2b98ac09d43149c5bcfd2c0080ea5379" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 40367 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 07:58:43.471382 2770 tablet_service.cc:1514] Processing DeleteTablet for tablet 18f0be6c9b944ce1b63600f456f272dd with delete_type TABLET_DATA_TOMBSTONED (TS 809d92b226a94aa59287ddc81c7ff77b not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:33210
I20240501 07:58:43.471885 5246 tablet_replica.cc:330] T 18f0be6c9b944ce1b63600f456f272dd P 809d92b226a94aa59287ddc81c7ff77b: stopping tablet replica
I20240501 07:58:43.472177 5246 raft_consensus.cc:2229] T 18f0be6c9b944ce1b63600f456f272dd P 809d92b226a94aa59287ddc81c7ff77b [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:43.472337 5246 raft_consensus.cc:2258] T 18f0be6c9b944ce1b63600f456f272dd P 809d92b226a94aa59287ddc81c7ff77b [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:43.473397 5246 ts_tablet_manager.cc:1875] T 18f0be6c9b944ce1b63600f456f272dd P 809d92b226a94aa59287ddc81c7ff77b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:58:43.475266 5246 ts_tablet_manager.cc:1888] T 18f0be6c9b944ce1b63600f456f272dd P 809d92b226a94aa59287ddc81c7ff77b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240501 07:58:43.475345 5246 log.cc:1196] T 18f0be6c9b944ce1b63600f456f272dd P 809d92b226a94aa59287ddc81c7ff77b: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/18f0be6c9b944ce1b63600f456f272dd
I20240501 07:58:43.475957 2636 catalog_manager.cc:4830] TS 809d92b226a94aa59287ddc81c7ff77b (127.0.99.129:33451): tablet 18f0be6c9b944ce1b63600f456f272dd (table rebalance_test_table_2 [id=82153012ad124c3398cc16ce35fb1b37]) successfully deleted
I20240501 07:58:43.484519 2922 raft_consensus.cc:1261] T ef28ebd14a684f1f9f5d6718bf8284b4 P 853826bd6abf488a8895392f3470a546 [term 2 FOLLOWER]: Refusing update from remote peer 9846630212094f49930ef629ee885368: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 07:58:43.485220 4771 consensus_queue.cc:1035] T ef28ebd14a684f1f9f5d6718bf8284b4 P 9846630212094f49930ef629ee885368 [LEADER]: Connected to new peer: Peer: permanent_uuid: "853826bd6abf488a8895392f3470a546" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39801 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 07:58:43.599021 2922 raft_consensus.cc:1261] T 102891a28f95444dac4798b2aace3857 P 853826bd6abf488a8895392f3470a546 [term 2 FOLLOWER]: Refusing update from remote peer 9846630212094f49930ef629ee885368: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 07:58:43.599484 4771 consensus_queue.cc:1035] T 102891a28f95444dac4798b2aace3857 P 9846630212094f49930ef629ee885368 [LEADER]: Connected to new peer: Peer: permanent_uuid: "853826bd6abf488a8895392f3470a546" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39801 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 07:58:43.651841 3911 consensus_queue.cc:237] T ef28ebd14a684f1f9f5d6718bf8284b4 P 9846630212094f49930ef629ee885368 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9846630212094f49930ef629ee885368" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33467 } attrs { promote: false } }
I20240501 07:58:43.652556 4771 raft_consensus.cc:2932] T ef28ebd14a684f1f9f5d6718bf8284b4 P 9846630212094f49930ef629ee885368 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 853826bd6abf488a8895392f3470a546 (127.0.99.130) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9846630212094f49930ef629ee885368" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33467 } attrs { promote: false } } }
I20240501 07:58:43.653895 2633 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ef28ebd14a684f1f9f5d6718bf8284b4 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240501 07:58:43.654914 2644 catalog_manager.cc:5484] T ef28ebd14a684f1f9f5d6718bf8284b4 P 9846630212094f49930ef629ee885368 reported cstate change: config changed from index 5 to 7, VOTER 853826bd6abf488a8895392f3470a546 (127.0.99.130) evicted. New cstate: current_term: 2 leader_uuid: "9846630212094f49930ef629ee885368" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9846630212094f49930ef629ee885368" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33467 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 07:58:43.657274 2901 tablet_service.cc:1514] Processing DeleteTablet for tablet ef28ebd14a684f1f9f5d6718bf8284b4 with delete_type TABLET_DATA_TOMBSTONED (TS 853826bd6abf488a8895392f3470a546 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:37258
I20240501 07:58:43.657799 5335 tablet_replica.cc:330] T ef28ebd14a684f1f9f5d6718bf8284b4 P 853826bd6abf488a8895392f3470a546: stopping tablet replica
I20240501 07:58:43.658066 5335 raft_consensus.cc:2229] T ef28ebd14a684f1f9f5d6718bf8284b4 P 853826bd6abf488a8895392f3470a546 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:43.658222 5335 raft_consensus.cc:2258] T ef28ebd14a684f1f9f5d6718bf8284b4 P 853826bd6abf488a8895392f3470a546 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:43.659164 5335 ts_tablet_manager.cc:1875] T ef28ebd14a684f1f9f5d6718bf8284b4 P 853826bd6abf488a8895392f3470a546: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:58:43.661118 5335 ts_tablet_manager.cc:1888] T ef28ebd14a684f1f9f5d6718bf8284b4 P 853826bd6abf488a8895392f3470a546: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240501 07:58:43.661201 5335 log.cc:1196] T ef28ebd14a684f1f9f5d6718bf8284b4 P 853826bd6abf488a8895392f3470a546: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/ef28ebd14a684f1f9f5d6718bf8284b4
I20240501 07:58:43.661787 2636 catalog_manager.cc:4830] TS 853826bd6abf488a8895392f3470a546 (127.0.99.130:39801): tablet ef28ebd14a684f1f9f5d6718bf8284b4 (table rebalance_test_table_4 [id=c04f011aac574a36aceb005abe2af590]) successfully deleted
I20240501 07:58:43.669409 4457 consensus_queue.cc:237] T fa6cf4447cfe49ecbbfe94accb24aa97 P 98526b95c3e94668a1673fe660495876 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "98526b95c3e94668a1673fe660495876" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39073 } attrs { promote: false } }
I20240501 07:58:43.670099 5160 raft_consensus.cc:2932] T fa6cf4447cfe49ecbbfe94accb24aa97 P 98526b95c3e94668a1673fe660495876 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 809d92b226a94aa59287ddc81c7ff77b (127.0.99.129) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "98526b95c3e94668a1673fe660495876" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39073 } attrs { promote: false } } }
I20240501 07:58:43.671496 2634 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet fa6cf4447cfe49ecbbfe94accb24aa97 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240501 07:58:43.672500 2644 catalog_manager.cc:5484] T fa6cf4447cfe49ecbbfe94accb24aa97 P 98526b95c3e94668a1673fe660495876 reported cstate change: config changed from index 5 to 7, VOTER 809d92b226a94aa59287ddc81c7ff77b (127.0.99.129) evicted. New cstate: current_term: 2 leader_uuid: "98526b95c3e94668a1673fe660495876" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "98526b95c3e94668a1673fe660495876" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39073 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 07:58:43.674908 2770 tablet_service.cc:1514] Processing DeleteTablet for tablet fa6cf4447cfe49ecbbfe94accb24aa97 with delete_type TABLET_DATA_TOMBSTONED (TS 809d92b226a94aa59287ddc81c7ff77b not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:33210
I20240501 07:58:43.675148 5246 tablet_replica.cc:330] T fa6cf4447cfe49ecbbfe94accb24aa97 P 809d92b226a94aa59287ddc81c7ff77b: stopping tablet replica
I20240501 07:58:43.675295 5246 raft_consensus.cc:2229] T fa6cf4447cfe49ecbbfe94accb24aa97 P 809d92b226a94aa59287ddc81c7ff77b [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:43.675417 5246 raft_consensus.cc:2258] T fa6cf4447cfe49ecbbfe94accb24aa97 P 809d92b226a94aa59287ddc81c7ff77b [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:43.676069 5246 ts_tablet_manager.cc:1875] T fa6cf4447cfe49ecbbfe94accb24aa97 P 809d92b226a94aa59287ddc81c7ff77b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:58:43.677860 5246 ts_tablet_manager.cc:1888] T fa6cf4447cfe49ecbbfe94accb24aa97 P 809d92b226a94aa59287ddc81c7ff77b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240501 07:58:43.677933 5246 log.cc:1196] T fa6cf4447cfe49ecbbfe94accb24aa97 P 809d92b226a94aa59287ddc81c7ff77b: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/fa6cf4447cfe49ecbbfe94accb24aa97
I20240501 07:58:43.678459 2636 catalog_manager.cc:4830] TS 809d92b226a94aa59287ddc81c7ff77b (127.0.99.129:33451): tablet fa6cf4447cfe49ecbbfe94accb24aa97 (table rebalance_test_table_0 [id=176edd724cff404e82ccb10fe1aed740]) successfully deleted
I20240501 07:58:43.704493 3911 consensus_queue.cc:237] T 102891a28f95444dac4798b2aace3857 P 9846630212094f49930ef629ee885368 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9846630212094f49930ef629ee885368" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33467 } attrs { promote: false } }
I20240501 07:58:43.705266 4771 raft_consensus.cc:2932] T 102891a28f95444dac4798b2aace3857 P 9846630212094f49930ef629ee885368 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 853826bd6abf488a8895392f3470a546 (127.0.99.130) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9846630212094f49930ef629ee885368" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33467 } attrs { promote: false } } }
I20240501 07:58:43.706552 2922 raft_consensus.cc:1261] T 0b2690f2189b43f9a45e1a90605623ad P 853826bd6abf488a8895392f3470a546 [term 2 FOLLOWER]: Refusing update from remote peer cb0a6f265d7e4362aea53c916ceb49ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 07:58:43.707020 2633 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 102891a28f95444dac4798b2aace3857 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240501 07:58:43.707255 4829 consensus_queue.cc:1035] T 0b2690f2189b43f9a45e1a90605623ad P cb0a6f265d7e4362aea53c916ceb49ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "853826bd6abf488a8895392f3470a546" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39801 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 07:58:43.707806 2644 catalog_manager.cc:5484] T 102891a28f95444dac4798b2aace3857 P 9846630212094f49930ef629ee885368 reported cstate change: config changed from index 5 to 7, VOTER 853826bd6abf488a8895392f3470a546 (127.0.99.130) evicted. New cstate: current_term: 2 leader_uuid: "9846630212094f49930ef629ee885368" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9846630212094f49930ef629ee885368" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 33467 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 07:58:43.711761 2901 tablet_service.cc:1514] Processing DeleteTablet for tablet 102891a28f95444dac4798b2aace3857 with delete_type TABLET_DATA_TOMBSTONED (TS 853826bd6abf488a8895392f3470a546 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:37258
I20240501 07:58:43.712334 5335 tablet_replica.cc:330] T 102891a28f95444dac4798b2aace3857 P 853826bd6abf488a8895392f3470a546: stopping tablet replica
I20240501 07:58:43.712491 5335 raft_consensus.cc:2229] T 102891a28f95444dac4798b2aace3857 P 853826bd6abf488a8895392f3470a546 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:43.712604 5335 raft_consensus.cc:2258] T 102891a28f95444dac4798b2aace3857 P 853826bd6abf488a8895392f3470a546 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:43.713263 5335 ts_tablet_manager.cc:1875] T 102891a28f95444dac4798b2aace3857 P 853826bd6abf488a8895392f3470a546: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:58:43.715027 5335 ts_tablet_manager.cc:1888] T 102891a28f95444dac4798b2aace3857 P 853826bd6abf488a8895392f3470a546: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240501 07:58:43.715124 5335 log.cc:1196] T 102891a28f95444dac4798b2aace3857 P 853826bd6abf488a8895392f3470a546: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/102891a28f95444dac4798b2aace3857
I20240501 07:58:43.715703 2636 catalog_manager.cc:4830] TS 853826bd6abf488a8895392f3470a546 (127.0.99.130:39801): tablet 102891a28f95444dac4798b2aace3857 (table rebalance_test_table_3 [id=30a898036c104f329727c3b9bc8b9100]) successfully deleted
I20240501 07:58:43.753679 4054 consensus_queue.cc:237] T 0b2690f2189b43f9a45e1a90605623ad P cb0a6f265d7e4362aea53c916ceb49ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "cb0a6f265d7e4362aea53c916ceb49ec" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43375 } attrs { promote: false } }
I20240501 07:58:43.754360 4829 raft_consensus.cc:2932] T 0b2690f2189b43f9a45e1a90605623ad P cb0a6f265d7e4362aea53c916ceb49ec [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 853826bd6abf488a8895392f3470a546 (127.0.99.130) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "cb0a6f265d7e4362aea53c916ceb49ec" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43375 } attrs { promote: false } } }
I20240501 07:58:43.755697 2633 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 0b2690f2189b43f9a45e1a90605623ad with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240501 07:58:43.756538 2641 catalog_manager.cc:5484] T 0b2690f2189b43f9a45e1a90605623ad P cb0a6f265d7e4362aea53c916ceb49ec reported cstate change: config changed from index 5 to 7, VOTER 853826bd6abf488a8895392f3470a546 (127.0.99.130) evicted. New cstate: current_term: 2 leader_uuid: "cb0a6f265d7e4362aea53c916ceb49ec" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "cb0a6f265d7e4362aea53c916ceb49ec" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43375 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 07:58:43.758934 2901 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b2690f2189b43f9a45e1a90605623ad with delete_type TABLET_DATA_TOMBSTONED (TS 853826bd6abf488a8895392f3470a546 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:37258
I20240501 07:58:43.759156 5335 tablet_replica.cc:330] T 0b2690f2189b43f9a45e1a90605623ad P 853826bd6abf488a8895392f3470a546: stopping tablet replica
I20240501 07:58:43.759265 5335 raft_consensus.cc:2229] T 0b2690f2189b43f9a45e1a90605623ad P 853826bd6abf488a8895392f3470a546 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:43.759351 5335 raft_consensus.cc:2258] T 0b2690f2189b43f9a45e1a90605623ad P 853826bd6abf488a8895392f3470a546 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:43.760002 5335 ts_tablet_manager.cc:1875] T 0b2690f2189b43f9a45e1a90605623ad P 853826bd6abf488a8895392f3470a546: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:58:43.761880 5335 ts_tablet_manager.cc:1888] T 0b2690f2189b43f9a45e1a90605623ad P 853826bd6abf488a8895392f3470a546: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240501 07:58:43.761946 5335 log.cc:1196] T 0b2690f2189b43f9a45e1a90605623ad P 853826bd6abf488a8895392f3470a546: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/0b2690f2189b43f9a45e1a90605623ad
I20240501 07:58:43.762391 2636 catalog_manager.cc:4830] TS 853826bd6abf488a8895392f3470a546 (127.0.99.130:39801): tablet 0b2690f2189b43f9a45e1a90605623ad (table rebalance_test_table_1 [id=8bdc688d20ef4b1d825861547a2f7334]) successfully deleted
I20240501 07:58:43.777078 2791 raft_consensus.cc:1261] T c64dae3606e049e2a72b00148fffc748 P 809d92b226a94aa59287ddc81c7ff77b [term 2 FOLLOWER]: Refusing update from remote peer 2471afee7ac94728a1506dc907ef8fbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 07:58:43.777614 5084 consensus_queue.cc:1035] T c64dae3606e049e2a72b00148fffc748 P 2471afee7ac94728a1506dc907ef8fbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "809d92b226a94aa59287ddc81c7ff77b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33451 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 07:58:43.781342 2791 raft_consensus.cc:1261] T 149f1f36b61a4d148c2003caaa7fce0c P 809d92b226a94aa59287ddc81c7ff77b [term 2 FOLLOWER]: Refusing update from remote peer cb0a6f265d7e4362aea53c916ceb49ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 07:58:43.781725 4829 consensus_queue.cc:1035] T 149f1f36b61a4d148c2003caaa7fce0c P cb0a6f265d7e4362aea53c916ceb49ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "809d92b226a94aa59287ddc81c7ff77b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33451 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 07:58:43.793299 2791 raft_consensus.cc:1261] T db9747650ebf4556ac30e4138821695b P 809d92b226a94aa59287ddc81c7ff77b [term 2 FOLLOWER]: Refusing update from remote peer 98526b95c3e94668a1673fe660495876: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 07:58:43.793737 4754 consensus_queue.cc:1035] T db9747650ebf4556ac30e4138821695b P 98526b95c3e94668a1673fe660495876 [LEADER]: Connected to new peer: Peer: permanent_uuid: "809d92b226a94aa59287ddc81c7ff77b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33451 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 07:58:43.800742 2922 raft_consensus.cc:1261] T c63214bc45b3424492cc114662f6839e P 853826bd6abf488a8895392f3470a546 [term 2 FOLLOWER]: Refusing update from remote peer 2471afee7ac94728a1506dc907ef8fbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 07:58:43.801540 5242 consensus_queue.cc:1035] T c63214bc45b3424492cc114662f6839e P 2471afee7ac94728a1506dc907ef8fbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "853826bd6abf488a8895392f3470a546" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39801 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 07:58:43.818336 4054 consensus_queue.cc:237] T 149f1f36b61a4d148c2003caaa7fce0c P cb0a6f265d7e4362aea53c916ceb49ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "cb0a6f265d7e4362aea53c916ceb49ec" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43375 } attrs { promote: false } }
I20240501 07:58:43.818985 4829 raft_consensus.cc:2932] T 149f1f36b61a4d148c2003caaa7fce0c P cb0a6f265d7e4362aea53c916ceb49ec [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 809d92b226a94aa59287ddc81c7ff77b (127.0.99.129) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "cb0a6f265d7e4362aea53c916ceb49ec" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43375 } attrs { promote: false } } }
I20240501 07:58:43.820140 2633 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 149f1f36b61a4d148c2003caaa7fce0c with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240501 07:58:43.821103 2644 catalog_manager.cc:5484] T 149f1f36b61a4d148c2003caaa7fce0c P cb0a6f265d7e4362aea53c916ceb49ec reported cstate change: config changed from index 5 to 7, VOTER 809d92b226a94aa59287ddc81c7ff77b (127.0.99.129) evicted. New cstate: current_term: 2 leader_uuid: "cb0a6f265d7e4362aea53c916ceb49ec" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "cb0a6f265d7e4362aea53c916ceb49ec" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43375 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 07:58:43.823439 2770 tablet_service.cc:1514] Processing DeleteTablet for tablet 149f1f36b61a4d148c2003caaa7fce0c with delete_type TABLET_DATA_TOMBSTONED (TS 809d92b226a94aa59287ddc81c7ff77b not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:33210
I20240501 07:58:43.823675 5246 tablet_replica.cc:330] T 149f1f36b61a4d148c2003caaa7fce0c P 809d92b226a94aa59287ddc81c7ff77b: stopping tablet replica
I20240501 07:58:43.823835 5246 raft_consensus.cc:2229] T 149f1f36b61a4d148c2003caaa7fce0c P 809d92b226a94aa59287ddc81c7ff77b [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:43.823944 5246 raft_consensus.cc:2258] T 149f1f36b61a4d148c2003caaa7fce0c P 809d92b226a94aa59287ddc81c7ff77b [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:43.824633 5246 ts_tablet_manager.cc:1875] T 149f1f36b61a4d148c2003caaa7fce0c P 809d92b226a94aa59287ddc81c7ff77b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:58:43.826654 5246 ts_tablet_manager.cc:1888] T 149f1f36b61a4d148c2003caaa7fce0c P 809d92b226a94aa59287ddc81c7ff77b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240501 07:58:43.826745 5246 log.cc:1196] T 149f1f36b61a4d148c2003caaa7fce0c P 809d92b226a94aa59287ddc81c7ff77b: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/149f1f36b61a4d148c2003caaa7fce0c
I20240501 07:58:43.827279 2636 catalog_manager.cc:4830] TS 809d92b226a94aa59287ddc81c7ff77b (127.0.99.129:33451): tablet 149f1f36b61a4d148c2003caaa7fce0c (table rebalance_test_table_3 [id=30a898036c104f329727c3b9bc8b9100]) successfully deleted
I20240501 07:58:43.905545 4457 consensus_queue.cc:237] T db9747650ebf4556ac30e4138821695b P 98526b95c3e94668a1673fe660495876 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "98526b95c3e94668a1673fe660495876" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39073 } attrs { promote: false } }
I20240501 07:58:43.906179 5160 raft_consensus.cc:2932] T db9747650ebf4556ac30e4138821695b P 98526b95c3e94668a1673fe660495876 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 809d92b226a94aa59287ddc81c7ff77b (127.0.99.129) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "98526b95c3e94668a1673fe660495876" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39073 } attrs { promote: false } } }
I20240501 07:58:43.907500 2634 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet db9747650ebf4556ac30e4138821695b with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240501 07:58:43.908494 2644 catalog_manager.cc:5484] T db9747650ebf4556ac30e4138821695b P 98526b95c3e94668a1673fe660495876 reported cstate change: config changed from index 5 to 7, VOTER 809d92b226a94aa59287ddc81c7ff77b (127.0.99.129) evicted. New cstate: current_term: 2 leader_uuid: "98526b95c3e94668a1673fe660495876" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "98526b95c3e94668a1673fe660495876" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39073 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 07:58:43.910764 2770 tablet_service.cc:1514] Processing DeleteTablet for tablet db9747650ebf4556ac30e4138821695b with delete_type TABLET_DATA_TOMBSTONED (TS 809d92b226a94aa59287ddc81c7ff77b not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:33210
I20240501 07:58:43.911007 5246 tablet_replica.cc:330] T db9747650ebf4556ac30e4138821695b P 809d92b226a94aa59287ddc81c7ff77b: stopping tablet replica
I20240501 07:58:43.911141 5246 raft_consensus.cc:2229] T db9747650ebf4556ac30e4138821695b P 809d92b226a94aa59287ddc81c7ff77b [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:43.911237 5246 raft_consensus.cc:2258] T db9747650ebf4556ac30e4138821695b P 809d92b226a94aa59287ddc81c7ff77b [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:43.911844 5246 ts_tablet_manager.cc:1875] T db9747650ebf4556ac30e4138821695b P 809d92b226a94aa59287ddc81c7ff77b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:58:43.913563 5246 ts_tablet_manager.cc:1888] T db9747650ebf4556ac30e4138821695b P 809d92b226a94aa59287ddc81c7ff77b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240501 07:58:43.913625 5246 log.cc:1196] T db9747650ebf4556ac30e4138821695b P 809d92b226a94aa59287ddc81c7ff77b: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/db9747650ebf4556ac30e4138821695b
I20240501 07:58:43.914093 2636 catalog_manager.cc:4830] TS 809d92b226a94aa59287ddc81c7ff77b (127.0.99.129:33451): tablet db9747650ebf4556ac30e4138821695b (table rebalance_test_table_1 [id=8bdc688d20ef4b1d825861547a2f7334]) successfully deleted
I20240501 07:58:43.958572 4326 consensus_queue.cc:237] T c64dae3606e049e2a72b00148fffc748 P 2471afee7ac94728a1506dc907ef8fbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2471afee7ac94728a1506dc907ef8fbb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38167 } attrs { promote: false } }
I20240501 07:58:43.959194 5242 raft_consensus.cc:2932] T c64dae3606e049e2a72b00148fffc748 P 2471afee7ac94728a1506dc907ef8fbb [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 809d92b226a94aa59287ddc81c7ff77b (127.0.99.129) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2471afee7ac94728a1506dc907ef8fbb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38167 } attrs { promote: false } } }
I20240501 07:58:43.960525 2636 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c64dae3606e049e2a72b00148fffc748 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240501 07:58:43.961405 2644 catalog_manager.cc:5484] T c64dae3606e049e2a72b00148fffc748 P 2471afee7ac94728a1506dc907ef8fbb reported cstate change: config changed from index 5 to 7, VOTER 809d92b226a94aa59287ddc81c7ff77b (127.0.99.129) evicted. New cstate: current_term: 2 leader_uuid: "2471afee7ac94728a1506dc907ef8fbb" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2471afee7ac94728a1506dc907ef8fbb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38167 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 07:58:43.963706 2770 tablet_service.cc:1514] Processing DeleteTablet for tablet c64dae3606e049e2a72b00148fffc748 with delete_type TABLET_DATA_TOMBSTONED (TS 809d92b226a94aa59287ddc81c7ff77b not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:33210
I20240501 07:58:43.963965 5246 tablet_replica.cc:330] T c64dae3606e049e2a72b00148fffc748 P 809d92b226a94aa59287ddc81c7ff77b: stopping tablet replica
I20240501 07:58:43.964073 5246 raft_consensus.cc:2229] T c64dae3606e049e2a72b00148fffc748 P 809d92b226a94aa59287ddc81c7ff77b [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:43.964160 5246 raft_consensus.cc:2258] T c64dae3606e049e2a72b00148fffc748 P 809d92b226a94aa59287ddc81c7ff77b [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:43.964782 5246 ts_tablet_manager.cc:1875] T c64dae3606e049e2a72b00148fffc748 P 809d92b226a94aa59287ddc81c7ff77b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:58:43.966605 5246 ts_tablet_manager.cc:1888] T c64dae3606e049e2a72b00148fffc748 P 809d92b226a94aa59287ddc81c7ff77b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240501 07:58:43.966672 5246 log.cc:1196] T c64dae3606e049e2a72b00148fffc748 P 809d92b226a94aa59287ddc81c7ff77b: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/c64dae3606e049e2a72b00148fffc748
I20240501 07:58:43.967247 2636 catalog_manager.cc:4830] TS 809d92b226a94aa59287ddc81c7ff77b (127.0.99.129:33451): tablet c64dae3606e049e2a72b00148fffc748 (table rebalance_test_table_4 [id=c04f011aac574a36aceb005abe2af590]) successfully deleted
I20240501 07:58:44.041664 4326 consensus_queue.cc:237] T c63214bc45b3424492cc114662f6839e P 2471afee7ac94728a1506dc907ef8fbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2471afee7ac94728a1506dc907ef8fbb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38167 } attrs { promote: false } }
I20240501 07:58:44.042311 5242 raft_consensus.cc:2932] T c63214bc45b3424492cc114662f6839e P 2471afee7ac94728a1506dc907ef8fbb [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 853826bd6abf488a8895392f3470a546 (127.0.99.130) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2471afee7ac94728a1506dc907ef8fbb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38167 } attrs { promote: false } } }
I20240501 07:58:44.043514 2636 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c63214bc45b3424492cc114662f6839e with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240501 07:58:44.044405 2644 catalog_manager.cc:5484] T c63214bc45b3424492cc114662f6839e P 2471afee7ac94728a1506dc907ef8fbb reported cstate change: config changed from index 5 to 7, VOTER 853826bd6abf488a8895392f3470a546 (127.0.99.130) evicted. New cstate: current_term: 2 leader_uuid: "2471afee7ac94728a1506dc907ef8fbb" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2471afee7ac94728a1506dc907ef8fbb" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 38167 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 07:58:44.046756 2901 tablet_service.cc:1514] Processing DeleteTablet for tablet c63214bc45b3424492cc114662f6839e with delete_type TABLET_DATA_TOMBSTONED (TS 853826bd6abf488a8895392f3470a546 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:37258
I20240501 07:58:44.046993 5335 tablet_replica.cc:330] T c63214bc45b3424492cc114662f6839e P 853826bd6abf488a8895392f3470a546: stopping tablet replica
I20240501 07:58:44.047132 5335 raft_consensus.cc:2229] T c63214bc45b3424492cc114662f6839e P 853826bd6abf488a8895392f3470a546 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:44.047240 5335 raft_consensus.cc:2258] T c63214bc45b3424492cc114662f6839e P 853826bd6abf488a8895392f3470a546 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:44.047834 5335 ts_tablet_manager.cc:1875] T c63214bc45b3424492cc114662f6839e P 853826bd6abf488a8895392f3470a546: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:58:44.049556 5335 ts_tablet_manager.cc:1888] T c63214bc45b3424492cc114662f6839e P 853826bd6abf488a8895392f3470a546: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240501 07:58:44.049623 5335 log.cc:1196] T c63214bc45b3424492cc114662f6839e P 853826bd6abf488a8895392f3470a546: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/c63214bc45b3424492cc114662f6839e
I20240501 07:58:44.050127 2636 catalog_manager.cc:4830] TS 853826bd6abf488a8895392f3470a546 (127.0.99.130:39801): tablet c63214bc45b3424492cc114662f6839e (table rebalance_test_table_0 [id=176edd724cff404e82ccb10fe1aed740]) successfully deleted
I20240501 07:58:44.150022 4306 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:44.154395 3903 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:44.155617 4166 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:44.158080 4437 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:44.165285 2901 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 07:58:44.166774 2770 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 07:58:44.167397 4034 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:44.257107 4166 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:44.260756 2770 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 07:58:44.262395 3903 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:44.268226 4034 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:44.268512 2901 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 07:58:44.268850 4437 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:44.274988 4306 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:44.531941 2901 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 07:58:44.532351 4306 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:44.532933 4437 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:44.532999 3903 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:44.535102 4166 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:44.536348 2770 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 07:58:44.546571 4034 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:44.679419 2901 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 07:58:44.680222 2770 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 07:58:44.680814 4437 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:44.684530 4034 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:44.691696 4166 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:44.701241 4306 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:44.701814 3903 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:44.808398 2901 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 07:58:44.809208 4306 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:44.809455 2770 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 07:58:44.810519 3903 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:44.811311 4437 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:44.815078 4166 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:44.834416 4034 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
95f14cded5744864a8da22ac2b8b06fa | 127.0.99.190:42355 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
allow_unsafe_replication_factor | true | runtime,unsafe | all 1 server(s) checked
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.0.99.148:34309 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
2471afee7ac94728a1506dc907ef8fbb | 127.0.99.134:38167 | HEALTHY | <none> | 2 | 0
2b98ac09d43149c5bcfd2c0080ea5379 | 127.0.99.133:40367 | HEALTHY | <none> | 2 | 0
809d92b226a94aa59287ddc81c7ff77b | 127.0.99.129:33451 | HEALTHY | <none> | 3 | 0
853826bd6abf488a8895392f3470a546 | 127.0.99.130:39801 | HEALTHY | <none> | 3 | 0
9846630212094f49930ef629ee885368 | 127.0.99.131:33467 | HEALTHY | <none> | 2 | 0
98526b95c3e94668a1673fe660495876 | 127.0.99.135:39073 | HEALTHY | <none> | 2 | 0
cb0a6f265d7e4362aea53c916ceb49ec | 127.0.99.132:43375 | HEALTHY | <none> | 2 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 7
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 7 server(s) checked
local_ip_for_outbound_sockets | 127.0.99.129 | experimental | 127.0.99.129:33451
local_ip_for_outbound_sockets | 127.0.99.130 | experimental | 127.0.99.130:39801
local_ip_for_outbound_sockets | 127.0.99.131 | experimental | 127.0.99.131:33467
local_ip_for_outbound_sockets | 127.0.99.132 | experimental | 127.0.99.132:43375
local_ip_for_outbound_sockets | 127.0.99.133 | experimental | 127.0.99.133:40367
local_ip_for_outbound_sockets | 127.0.99.134 | experimental | 127.0.99.134:38167
local_ip_for_outbound_sockets | 127.0.99.135 | experimental | 127.0.99.135:39073
never_fsync | true | unsafe,advanced | all 7 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 7 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 7 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 7 server(s) checked
server_dump_info_format | pb | hidden | all 7 server(s) checked
server_dump_info_path | /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.0.99.129:33451
server_dump_info_path | /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.0.99.130:39801
server_dump_info_path | /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.0.99.131:33467
server_dump_info_path | /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.0.99.132:43375
server_dump_info_path | /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.0.99.133:40367
server_dump_info_path | /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.0.99.134:38167
server_dump_info_path | /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden | 127.0.99.135:39073
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.0.99.148:34309 | all 7 server(s) checked
time_source | builtin | all 7 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 8 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 1 | HEALTHY | 1 | 1 | 0 | 0 | 0
rebalance_test_table_0 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_4 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 2
First Quartile | 2
Median | 2
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 7
Tables | 6
Tablets | 16
Replicas | 16
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240501 07:58:44.865563 398 log_verifier.cc:126] Checking tablet 0b2690f2189b43f9a45e1a90605623ad
I20240501 07:58:44.877123 398 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 0b2690f2189b43f9a45e1a90605623ad
I20240501 07:58:44.877214 398 log_verifier.cc:126] Checking tablet 102891a28f95444dac4798b2aace3857
I20240501 07:58:44.886416 398 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 102891a28f95444dac4798b2aace3857
I20240501 07:58:44.886487 398 log_verifier.cc:126] Checking tablet 149f1f36b61a4d148c2003caaa7fce0c
I20240501 07:58:44.896088 398 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 149f1f36b61a4d148c2003caaa7fce0c
I20240501 07:58:44.896158 398 log_verifier.cc:126] Checking tablet 14a68b0c3f984b999ddc7856ca076c8f
I20240501 07:58:44.905642 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 14a68b0c3f984b999ddc7856ca076c8f
I20240501 07:58:44.905706 398 log_verifier.cc:126] Checking tablet 18f0be6c9b944ce1b63600f456f272dd
I20240501 07:58:44.915711 398 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 18f0be6c9b944ce1b63600f456f272dd
I20240501 07:58:44.915777 398 log_verifier.cc:126] Checking tablet 243736c7fc354ccd86f3ac42401c61c0
I20240501 07:58:44.925354 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 243736c7fc354ccd86f3ac42401c61c0
I20240501 07:58:44.925424 398 log_verifier.cc:126] Checking tablet 3264671dc2714e98846bc54938cf358c
I20240501 07:58:44.934772 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 3264671dc2714e98846bc54938cf358c
I20240501 07:58:44.934840 398 log_verifier.cc:126] Checking tablet 3e1613b40cab4477952e3c04f2b16e5d
I20240501 07:58:44.944303 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 3e1613b40cab4477952e3c04f2b16e5d
I20240501 07:58:44.944379 398 log_verifier.cc:126] Checking tablet 444c036684fe4852ba6aaab77411e3a4
I20240501 07:58:44.953958 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 444c036684fe4852ba6aaab77411e3a4
I20240501 07:58:44.954017 398 log_verifier.cc:126] Checking tablet c4d49398b37446c383dc0be576a3263f
I20240501 07:58:44.962256 398 log_verifier.cc:177] Verified matching terms for 2 ops in tablet c4d49398b37446c383dc0be576a3263f
I20240501 07:58:44.962327 398 log_verifier.cc:126] Checking tablet c63214bc45b3424492cc114662f6839e
I20240501 07:58:44.971938 398 log_verifier.cc:177] Verified matching terms for 7 ops in tablet c63214bc45b3424492cc114662f6839e
I20240501 07:58:44.972011 398 log_verifier.cc:126] Checking tablet c64dae3606e049e2a72b00148fffc748
I20240501 07:58:44.981406 398 log_verifier.cc:177] Verified matching terms for 7 ops in tablet c64dae3606e049e2a72b00148fffc748
I20240501 07:58:44.981472 398 log_verifier.cc:126] Checking tablet db9747650ebf4556ac30e4138821695b
I20240501 07:58:44.990854 398 log_verifier.cc:177] Verified matching terms for 7 ops in tablet db9747650ebf4556ac30e4138821695b
I20240501 07:58:44.990932 398 log_verifier.cc:126] Checking tablet ef28ebd14a684f1f9f5d6718bf8284b4
I20240501 07:58:45.000418 398 log_verifier.cc:177] Verified matching terms for 7 ops in tablet ef28ebd14a684f1f9f5d6718bf8284b4
I20240501 07:58:45.000487 398 log_verifier.cc:126] Checking tablet f4277f2f67dd492297a8b36edc366bba
I20240501 07:58:45.010783 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet f4277f2f67dd492297a8b36edc366bba
I20240501 07:58:45.010861 398 log_verifier.cc:126] Checking tablet fa6cf4447cfe49ecbbfe94accb24aa97
I20240501 07:58:45.021641 398 log_verifier.cc:177] Verified matching terms for 7 ops in tablet fa6cf4447cfe49ecbbfe94accb24aa97
I20240501 07:58:45.023835 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tbgEt/build/debug/bin/kudu
/tmp/dist-test-task7tbgEt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.136:0
--local_ip_for_outbound_sockets=127.0.99.136
--webserver_interface=127.0.99.136
--webserver_port=0
--tserver_master_addrs=127.0.99.190:42355
--builtin_ntp_servers=127.0.99.148:34309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240501 07:58:45.137032 5586 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:58:45.137290 5586 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 07:58:45.137354 5586 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:58:45.140985 5586 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 07:58:45.141068 5586 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 07:58:45.141141 5586 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.136
I20240501 07:58:45.144940 5586 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:34309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.136:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.0.99.136
--webserver_port=0
--tserver_master_addrs=127.0.99.190:42355
--never_fsync=true
--heap_profile_path=/tmp/kudu.5586
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.136
--log_dir=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 07:43:16 UTC on 1a15035a6830
build id 69
I20240501 07:58:45.146013 5586 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 07:58:45.147110 5586 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:58:45.156138 5592 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:58:45.157130 5591 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:58:45.157894 5594 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:58:45.159086 5586 server_base.cc:1034] running on GCE node
I20240501 07:58:45.159530 5586 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 07:58:45.160154 5586 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 07:58:45.161325 5586 hybrid_clock.cc:648] HybridClock initialized: now 1714550325161296 us; error 50 us; skew 500 ppm
I20240501 07:58:45.163514 5586 webserver.cc:460] Webserver started at http://127.0.99.136:41485/ using document root <none> and password file <none>
I20240501 07:58:45.164131 5586 fs_manager.cc:359] Metadata directory not provided
I20240501 07:58:45.164193 5586 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:58:45.164386 5586 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 07:58:45.166913 5586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "da0b6d111b69425585bd4008fda6771e"
format_stamp: "Formatted at 2024-05-01 07:58:45 on dist-test-slave-40g1"
I20240501 07:58:45.167694 5586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "da0b6d111b69425585bd4008fda6771e"
format_stamp: "Formatted at 2024-05-01 07:58:45 on dist-test-slave-40g1"
I20240501 07:58:45.173220 5586 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20240501 07:58:45.175686 5600 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:45.176923 5586 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 07:58:45.177067 5586 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "da0b6d111b69425585bd4008fda6771e"
format_stamp: "Formatted at 2024-05-01 07:58:45 on dist-test-slave-40g1"
I20240501 07:58:45.177178 5586 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:58:45.197891 5586 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:58:45.198583 5586 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 07:58:45.198748 5586 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:58:45.199288 5586 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 07:58:45.200186 5586 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 07:58:45.200235 5586 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:45.200295 5586 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 07:58:45.200322 5586 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:45.211069 5586 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.136:43521
I20240501 07:58:45.211082 5713 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.136:43521 every 8 connection(s)
I20240501 07:58:45.212100 5586 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240501 07:58:45.220161 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tbgEt/build/debug/bin/kudu as pid 5586
I20240501 07:58:45.220269 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240501 07:58:45.223500 5714 heartbeater.cc:347] Connected to a master server at 127.0.99.190:42355
I20240501 07:58:45.223798 5714 heartbeater.cc:464] Registering TS with master...
I20240501 07:58:45.224603 5714 heartbeater.cc:511] Master 127.0.99.190:42355 requested a full tablet report, sending...
I20240501 07:58:45.225742 2640 ts_manager.cc:194] Registered new tserver with Master: da0b6d111b69425585bd4008fda6771e (127.0.99.136:43521)
I20240501 07:58:45.226346 2640 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.136:41993
I20240501 07:58:45.428510 4306 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:45.428604 4166 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:45.432188 4437 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:45.439401 2770 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 07:58:45.441512 5648 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:45.447440 4034 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:45.457844 3903 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:45.463654 2901 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 07:58:45.586257 2901 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 07:58:45.586462 4306 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:45.588102 4166 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:45.588832 2770 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 07:58:45.591534 3903 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:45.592164 4034 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:45.601523 5648 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:58:45.602262 4437 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:45.647751 2922 consensus_queue.cc:237] T 3264671dc2714e98846bc54938cf358c P 853826bd6abf488a8895392f3470a546 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "853826bd6abf488a8895392f3470a546" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39801 } attrs { replace: true } } peers { permanent_uuid: "da0b6d111b69425585bd4008fda6771e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43521 } attrs { promote: true } }
I20240501 07:58:45.648752 5807 raft_consensus.cc:2932] T 3264671dc2714e98846bc54938cf358c P 853826bd6abf488a8895392f3470a546 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER da0b6d111b69425585bd4008fda6771e (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "853826bd6abf488a8895392f3470a546" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39801 } attrs { replace: true } } peers { permanent_uuid: "da0b6d111b69425585bd4008fda6771e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43521 } attrs { promote: true } } }
I20240501 07:58:45.651897 2649 catalog_manager.cc:5484] T 3264671dc2714e98846bc54938cf358c P 853826bd6abf488a8895392f3470a546 reported cstate change: config changed from index -1 to 4, NON_VOTER da0b6d111b69425585bd4008fda6771e (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "853826bd6abf488a8895392f3470a546" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "853826bd6abf488a8895392f3470a546" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39801 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "da0b6d111b69425585bd4008fda6771e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43521 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240501 07:58:45.653632 2856 consensus_peers.cc:485] T 3264671dc2714e98846bc54938cf358c P 853826bd6abf488a8895392f3470a546 -> Peer da0b6d111b69425585bd4008fda6771e (127.0.99.136:43521): Couldn't send request to peer da0b6d111b69425585bd4008fda6771e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3264671dc2714e98846bc54938cf358c. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:58:45.659198 2791 consensus_queue.cc:237] T 243736c7fc354ccd86f3ac42401c61c0 P 809d92b226a94aa59287ddc81c7ff77b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "809d92b226a94aa59287ddc81c7ff77b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33451 } attrs { replace: true } } peers { permanent_uuid: "da0b6d111b69425585bd4008fda6771e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43521 } attrs { promote: true } }
I20240501 07:58:45.660104 5820 raft_consensus.cc:2932] T 243736c7fc354ccd86f3ac42401c61c0 P 809d92b226a94aa59287ddc81c7ff77b [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER da0b6d111b69425585bd4008fda6771e (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "809d92b226a94aa59287ddc81c7ff77b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33451 } attrs { replace: true } } peers { permanent_uuid: "da0b6d111b69425585bd4008fda6771e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43521 } attrs { promote: true } } }
I20240501 07:58:45.662945 2649 catalog_manager.cc:5484] T 243736c7fc354ccd86f3ac42401c61c0 P 809d92b226a94aa59287ddc81c7ff77b reported cstate change: config changed from index -1 to 4, NON_VOTER da0b6d111b69425585bd4008fda6771e (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "809d92b226a94aa59287ddc81c7ff77b" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "809d92b226a94aa59287ddc81c7ff77b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33451 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "da0b6d111b69425585bd4008fda6771e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43521 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240501 07:58:45.664774 2725 consensus_peers.cc:485] T 243736c7fc354ccd86f3ac42401c61c0 P 809d92b226a94aa59287ddc81c7ff77b -> Peer da0b6d111b69425585bd4008fda6771e (127.0.99.136:43521): Couldn't send request to peer da0b6d111b69425585bd4008fda6771e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 243736c7fc354ccd86f3ac42401c61c0. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:58:46.184505 5854 ts_tablet_manager.cc:902] T 243736c7fc354ccd86f3ac42401c61c0 P da0b6d111b69425585bd4008fda6771e: Initiating tablet copy from peer 809d92b226a94aa59287ddc81c7ff77b (127.0.99.129:33451)
I20240501 07:58:46.185279 5854 tablet_copy_client.cc:308] T 243736c7fc354ccd86f3ac42401c61c0 P da0b6d111b69425585bd4008fda6771e: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:33451
I20240501 07:58:46.188889 2811 tablet_copy_service.cc:140] P 809d92b226a94aa59287ddc81c7ff77b: Received BeginTabletCopySession request for tablet 243736c7fc354ccd86f3ac42401c61c0 from peer da0b6d111b69425585bd4008fda6771e ({username='slave'} at 127.0.99.136:59227)
I20240501 07:58:46.189337 2811 tablet_copy_service.cc:161] P 809d92b226a94aa59287ddc81c7ff77b: Beginning new tablet copy session on tablet 243736c7fc354ccd86f3ac42401c61c0 from peer da0b6d111b69425585bd4008fda6771e at {username='slave'} at 127.0.99.136:59227: session id = da0b6d111b69425585bd4008fda6771e-243736c7fc354ccd86f3ac42401c61c0
I20240501 07:58:46.190217 2811 tablet_copy_source_session.cc:205] T 243736c7fc354ccd86f3ac42401c61c0 P 809d92b226a94aa59287ddc81c7ff77b: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:58:46.193428 5854 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 243736c7fc354ccd86f3ac42401c61c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:46.198714 5854 tablet_copy_client.cc:783] T 243736c7fc354ccd86f3ac42401c61c0 P da0b6d111b69425585bd4008fda6771e: tablet copy: Starting download of 0 data blocks...
I20240501 07:58:46.198916 5854 tablet_copy_client.cc:647] T 243736c7fc354ccd86f3ac42401c61c0 P da0b6d111b69425585bd4008fda6771e: tablet copy: Starting download of 1 WAL segments...
I20240501 07:58:46.200754 5854 tablet_copy_client.cc:523] T 243736c7fc354ccd86f3ac42401c61c0 P da0b6d111b69425585bd4008fda6771e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:58:46.208359 5854 tablet_bootstrap.cc:492] T 243736c7fc354ccd86f3ac42401c61c0 P da0b6d111b69425585bd4008fda6771e: Bootstrap starting.
I20240501 07:58:46.214494 5854 log.cc:824] T 243736c7fc354ccd86f3ac42401c61c0 P da0b6d111b69425585bd4008fda6771e: Log is configured to *not* fsync() on all Append() calls
I20240501 07:58:46.229086 5714 heartbeater.cc:503] Master 127.0.99.190:42355 was elected leader, sending a full tablet report...
I20240501 07:58:46.229935 5862 ts_tablet_manager.cc:902] T 3264671dc2714e98846bc54938cf358c P da0b6d111b69425585bd4008fda6771e: Initiating tablet copy from peer 853826bd6abf488a8895392f3470a546 (127.0.99.130:39801)
I20240501 07:58:46.230487 5862 tablet_copy_client.cc:308] T 3264671dc2714e98846bc54938cf358c P da0b6d111b69425585bd4008fda6771e: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:39801
I20240501 07:58:46.234190 2941 tablet_copy_service.cc:140] P 853826bd6abf488a8895392f3470a546: Received BeginTabletCopySession request for tablet 3264671dc2714e98846bc54938cf358c from peer da0b6d111b69425585bd4008fda6771e ({username='slave'} at 127.0.99.136:39905)
I20240501 07:58:46.234297 2941 tablet_copy_service.cc:161] P 853826bd6abf488a8895392f3470a546: Beginning new tablet copy session on tablet 3264671dc2714e98846bc54938cf358c from peer da0b6d111b69425585bd4008fda6771e at {username='slave'} at 127.0.99.136:39905: session id = da0b6d111b69425585bd4008fda6771e-3264671dc2714e98846bc54938cf358c
I20240501 07:58:46.235019 2941 tablet_copy_source_session.cc:205] T 3264671dc2714e98846bc54938cf358c P 853826bd6abf488a8895392f3470a546: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:58:46.235985 5862 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3264671dc2714e98846bc54938cf358c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:46.237322 5854 tablet_bootstrap.cc:492] T 243736c7fc354ccd86f3ac42401c61c0 P da0b6d111b69425585bd4008fda6771e: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:58:46.237676 5854 tablet_bootstrap.cc:492] T 243736c7fc354ccd86f3ac42401c61c0 P da0b6d111b69425585bd4008fda6771e: Bootstrap complete.
I20240501 07:58:46.237944 5862 tablet_copy_client.cc:783] T 3264671dc2714e98846bc54938cf358c P da0b6d111b69425585bd4008fda6771e: tablet copy: Starting download of 0 data blocks...
I20240501 07:58:46.237983 5854 ts_tablet_manager.cc:1367] T 243736c7fc354ccd86f3ac42401c61c0 P da0b6d111b69425585bd4008fda6771e: Time spent bootstrapping tablet: real 0.030s user 0.030s sys 0.000s
I20240501 07:58:46.238103 5862 tablet_copy_client.cc:647] T 3264671dc2714e98846bc54938cf358c P da0b6d111b69425585bd4008fda6771e: tablet copy: Starting download of 1 WAL segments...
I20240501 07:58:46.239331 5862 tablet_copy_client.cc:523] T 3264671dc2714e98846bc54938cf358c P da0b6d111b69425585bd4008fda6771e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:58:46.240337 5854 raft_consensus.cc:348] T 243736c7fc354ccd86f3ac42401c61c0 P da0b6d111b69425585bd4008fda6771e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "809d92b226a94aa59287ddc81c7ff77b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33451 } attrs { replace: true } } peers { permanent_uuid: "da0b6d111b69425585bd4008fda6771e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43521 } attrs { promote: true } }
I20240501 07:58:46.240550 5854 raft_consensus.cc:729] T 243736c7fc354ccd86f3ac42401c61c0 P da0b6d111b69425585bd4008fda6771e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: da0b6d111b69425585bd4008fda6771e, State: Initialized, Role: LEARNER
I20240501 07:58:46.240562 5862 tablet_bootstrap.cc:492] T 3264671dc2714e98846bc54938cf358c P da0b6d111b69425585bd4008fda6771e: Bootstrap starting.
I20240501 07:58:46.240871 5854 consensus_queue.cc:260] T 243736c7fc354ccd86f3ac42401c61c0 P da0b6d111b69425585bd4008fda6771e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "809d92b226a94aa59287ddc81c7ff77b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33451 } attrs { replace: true } } peers { permanent_uuid: "da0b6d111b69425585bd4008fda6771e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43521 } attrs { promote: true } }
I20240501 07:58:46.241777 5854 ts_tablet_manager.cc:1398] T 243736c7fc354ccd86f3ac42401c61c0 P da0b6d111b69425585bd4008fda6771e: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20240501 07:58:46.242415 2811 tablet_copy_service.cc:342] P 809d92b226a94aa59287ddc81c7ff77b: Request end of tablet copy session da0b6d111b69425585bd4008fda6771e-243736c7fc354ccd86f3ac42401c61c0 received from {username='slave'} at 127.0.99.136:59227
I20240501 07:58:46.242496 2811 tablet_copy_service.cc:433] P 809d92b226a94aa59287ddc81c7ff77b: ending tablet copy session da0b6d111b69425585bd4008fda6771e-243736c7fc354ccd86f3ac42401c61c0 on tablet 243736c7fc354ccd86f3ac42401c61c0 with peer da0b6d111b69425585bd4008fda6771e
I20240501 07:58:46.258775 5862 tablet_bootstrap.cc:492] T 3264671dc2714e98846bc54938cf358c P da0b6d111b69425585bd4008fda6771e: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:58:46.259146 5862 tablet_bootstrap.cc:492] T 3264671dc2714e98846bc54938cf358c P da0b6d111b69425585bd4008fda6771e: Bootstrap complete.
I20240501 07:58:46.259346 5862 ts_tablet_manager.cc:1367] T 3264671dc2714e98846bc54938cf358c P da0b6d111b69425585bd4008fda6771e: Time spent bootstrapping tablet: real 0.019s user 0.016s sys 0.002s
I20240501 07:58:46.259897 5862 raft_consensus.cc:348] T 3264671dc2714e98846bc54938cf358c P da0b6d111b69425585bd4008fda6771e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "853826bd6abf488a8895392f3470a546" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39801 } attrs { replace: true } } peers { permanent_uuid: "da0b6d111b69425585bd4008fda6771e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43521 } attrs { promote: true } }
I20240501 07:58:46.260026 5862 raft_consensus.cc:729] T 3264671dc2714e98846bc54938cf358c P da0b6d111b69425585bd4008fda6771e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: da0b6d111b69425585bd4008fda6771e, State: Initialized, Role: LEARNER
I20240501 07:58:46.260138 5862 consensus_queue.cc:260] T 3264671dc2714e98846bc54938cf358c P da0b6d111b69425585bd4008fda6771e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "853826bd6abf488a8895392f3470a546" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39801 } attrs { replace: true } } peers { permanent_uuid: "da0b6d111b69425585bd4008fda6771e" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43521 } attrs { promote: true } }
I20240501 07:58:46.260413 5862 ts_tablet_manager.cc:1398] T 3264671dc2714e98846bc54938cf358c P da0b6d111b69425585bd4008fda6771e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:58:46.261062 2941 tablet_copy_service.cc:342] P 853826bd6abf488a8895392f3470a546: Request end of tablet copy session da0b6d111b69425585bd4008fda6771e-3264671dc2714e98846bc54938cf358c received from {username='slave'} at 127.0.99.136:39905
I20240501 07:58:46.261150 2941 tablet_copy_service.cc:433] P 853826bd6abf488a8895392f3470a546: ending tablet copy session da0b6d111b69425585bd4008fda6771e-3264671dc2714e98846bc54938cf358c on tablet 3264671dc2714e98846bc54938cf358c with peer da0b6d111b69425585bd4008fda6771e
I20240501 07:58:46.777040 5661 raft_consensus.cc:1203] T 243736c7fc354ccd86f3ac42401c61c0 P da0b6d111b69425585bd4008fda6771e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 07:58:46.839078 5661 raft_consensus.cc:1203] T 3264671dc2714e98846bc54938cf358c P da0b6d111b69425585bd4008fda6771e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 07:58:47.228358 5900 raft_consensus.cc:1050] T 243736c7fc354ccd86f3ac42401c61c0 P 809d92b226a94aa59287ddc81c7ff77b: attempting to promote NON_VOTER da0b6d111b69425585bd4008fda6771e to VOTER
I20240501 07:58:47.228780 5900 consensus_queue.cc:237] T 243736c7fc354ccd86f3ac42401c61c0 P 809d92b226a94aa59287ddc81c7ff77b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "809d92b226a94aa59287ddc81c7ff77b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33451 } attrs { replace: true } } peers { permanent_uuid: "da0b6d111b69425585bd4008fda6771e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43521 } attrs { promote: false } }
I20240501 07:58:47.230082 5661 raft_consensus.cc:1261] T 243736c7fc354ccd86f3ac42401c61c0 P da0b6d111b69425585bd4008fda6771e [term 1 LEARNER]: Refusing update from remote peer 809d92b226a94aa59287ddc81c7ff77b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:58:47.230552 5900 consensus_queue.cc:1035] T 243736c7fc354ccd86f3ac42401c61c0 P 809d92b226a94aa59287ddc81c7ff77b [LEADER]: Connected to new peer: Peer: permanent_uuid: "da0b6d111b69425585bd4008fda6771e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43521 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:58:47.233249 5899 raft_consensus.cc:2932] T 243736c7fc354ccd86f3ac42401c61c0 P 809d92b226a94aa59287ddc81c7ff77b [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, da0b6d111b69425585bd4008fda6771e (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "809d92b226a94aa59287ddc81c7ff77b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33451 } attrs { replace: true } } peers { permanent_uuid: "da0b6d111b69425585bd4008fda6771e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43521 } attrs { promote: false } } }
I20240501 07:58:47.233798 5661 raft_consensus.cc:2932] T 243736c7fc354ccd86f3ac42401c61c0 P da0b6d111b69425585bd4008fda6771e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, da0b6d111b69425585bd4008fda6771e (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "809d92b226a94aa59287ddc81c7ff77b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33451 } attrs { replace: true } } peers { permanent_uuid: "da0b6d111b69425585bd4008fda6771e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43521 } attrs { promote: false } } }
I20240501 07:58:47.235703 2644 catalog_manager.cc:5484] T 243736c7fc354ccd86f3ac42401c61c0 P 809d92b226a94aa59287ddc81c7ff77b reported cstate change: config changed from index 4 to 5, da0b6d111b69425585bd4008fda6771e (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "809d92b226a94aa59287ddc81c7ff77b" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "809d92b226a94aa59287ddc81c7ff77b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33451 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "da0b6d111b69425585bd4008fda6771e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43521 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 07:58:47.295850 5902 raft_consensus.cc:1050] T 3264671dc2714e98846bc54938cf358c P 853826bd6abf488a8895392f3470a546: attempting to promote NON_VOTER da0b6d111b69425585bd4008fda6771e to VOTER
I20240501 07:58:47.296209 5902 consensus_queue.cc:237] T 3264671dc2714e98846bc54938cf358c P 853826bd6abf488a8895392f3470a546 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "853826bd6abf488a8895392f3470a546" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39801 } attrs { replace: true } } peers { permanent_uuid: "da0b6d111b69425585bd4008fda6771e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43521 } attrs { promote: false } }
I20240501 07:58:47.297366 5661 raft_consensus.cc:1261] T 3264671dc2714e98846bc54938cf358c P da0b6d111b69425585bd4008fda6771e [term 1 LEARNER]: Refusing update from remote peer 853826bd6abf488a8895392f3470a546: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:58:47.297755 5902 consensus_queue.cc:1035] T 3264671dc2714e98846bc54938cf358c P 853826bd6abf488a8895392f3470a546 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da0b6d111b69425585bd4008fda6771e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43521 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:58:47.299623 5903 raft_consensus.cc:2932] T 3264671dc2714e98846bc54938cf358c P 853826bd6abf488a8895392f3470a546 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, da0b6d111b69425585bd4008fda6771e (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "853826bd6abf488a8895392f3470a546" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39801 } attrs { replace: true } } peers { permanent_uuid: "da0b6d111b69425585bd4008fda6771e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43521 } attrs { promote: false } } }
I20240501 07:58:47.300124 5661 raft_consensus.cc:2932] T 3264671dc2714e98846bc54938cf358c P da0b6d111b69425585bd4008fda6771e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, da0b6d111b69425585bd4008fda6771e (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "853826bd6abf488a8895392f3470a546" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39801 } attrs { replace: true } } peers { permanent_uuid: "da0b6d111b69425585bd4008fda6771e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43521 } attrs { promote: false } } }
I20240501 07:58:47.302089 2649 catalog_manager.cc:5484] T 3264671dc2714e98846bc54938cf358c P 853826bd6abf488a8895392f3470a546 reported cstate change: config changed from index 4 to 5, da0b6d111b69425585bd4008fda6771e (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "853826bd6abf488a8895392f3470a546" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "853826bd6abf488a8895392f3470a546" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39801 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "da0b6d111b69425585bd4008fda6771e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43521 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 07:58:47.374259 2791 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "243736c7fc354ccd86f3ac42401c61c0"
dest_uuid: "809d92b226a94aa59287ddc81c7ff77b"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:59702
I20240501 07:58:47.374414 2791 raft_consensus.cc:595] T 243736c7fc354ccd86f3ac42401c61c0 P 809d92b226a94aa59287ddc81c7ff77b [term 1 LEADER]: Received request to transfer leadership
I20240501 07:58:47.385174 2922 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3264671dc2714e98846bc54938cf358c"
dest_uuid: "853826bd6abf488a8895392f3470a546"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:43000
I20240501 07:58:47.385325 2922 raft_consensus.cc:595] T 3264671dc2714e98846bc54938cf358c P 853826bd6abf488a8895392f3470a546 [term 1 LEADER]: Received request to transfer leadership
I20240501 07:58:47.596521 2791 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "243736c7fc354ccd86f3ac42401c61c0"
dest_uuid: "809d92b226a94aa59287ddc81c7ff77b"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:59706
I20240501 07:58:47.596679 2791 raft_consensus.cc:595] T 243736c7fc354ccd86f3ac42401c61c0 P 809d92b226a94aa59287ddc81c7ff77b [term 1 LEADER]: Received request to transfer leadership
I20240501 07:58:47.607556 2922 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3264671dc2714e98846bc54938cf358c"
dest_uuid: "853826bd6abf488a8895392f3470a546"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:43028
I20240501 07:58:47.607707 2922 raft_consensus.cc:595] T 3264671dc2714e98846bc54938cf358c P 853826bd6abf488a8895392f3470a546 [term 1 LEADER]: Received request to transfer leadership
I20240501 07:58:47.721287 5899 raft_consensus.cc:979] T 243736c7fc354ccd86f3ac42401c61c0 P 809d92b226a94aa59287ddc81c7ff77b: : Instructing follower da0b6d111b69425585bd4008fda6771e to start an election
I20240501 07:58:47.721431 5899 raft_consensus.cc:1067] T 243736c7fc354ccd86f3ac42401c61c0 P 809d92b226a94aa59287ddc81c7ff77b [term 1 LEADER]: Signalling peer da0b6d111b69425585bd4008fda6771e to start an election
I20240501 07:58:47.721904 5661 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "243736c7fc354ccd86f3ac42401c61c0"
dest_uuid: "da0b6d111b69425585bd4008fda6771e"
from {username='slave'} at 127.0.99.129:53563
I20240501 07:58:47.722095 5661 raft_consensus.cc:482] T 243736c7fc354ccd86f3ac42401c61c0 P da0b6d111b69425585bd4008fda6771e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:58:47.722203 5661 raft_consensus.cc:3037] T 243736c7fc354ccd86f3ac42401c61c0 P da0b6d111b69425585bd4008fda6771e [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:58:47.723047 5661 raft_consensus.cc:504] T 243736c7fc354ccd86f3ac42401c61c0 P da0b6d111b69425585bd4008fda6771e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "809d92b226a94aa59287ddc81c7ff77b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33451 } attrs { replace: true } } peers { permanent_uuid: "da0b6d111b69425585bd4008fda6771e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43521 } attrs { promote: false } }
I20240501 07:58:47.723938 5661 leader_election.cc:288] T 243736c7fc354ccd86f3ac42401c61c0 P da0b6d111b69425585bd4008fda6771e [CANDIDATE]: Term 2 election: Requested vote from peers 809d92b226a94aa59287ddc81c7ff77b (127.0.99.129:33451)
I20240501 07:58:47.724339 2791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "243736c7fc354ccd86f3ac42401c61c0" candidate_uuid: "da0b6d111b69425585bd4008fda6771e" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "809d92b226a94aa59287ddc81c7ff77b"
I20240501 07:58:47.724462 2791 raft_consensus.cc:3032] T 243736c7fc354ccd86f3ac42401c61c0 P 809d92b226a94aa59287ddc81c7ff77b [term 1 LEADER]: Stepping down as leader of term 1
I20240501 07:58:47.724512 2791 raft_consensus.cc:729] T 243736c7fc354ccd86f3ac42401c61c0 P 809d92b226a94aa59287ddc81c7ff77b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 809d92b226a94aa59287ddc81c7ff77b, State: Running, Role: LEADER
I20240501 07:58:47.724655 2791 consensus_queue.cc:260] T 243736c7fc354ccd86f3ac42401c61c0 P 809d92b226a94aa59287ddc81c7ff77b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "809d92b226a94aa59287ddc81c7ff77b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33451 } attrs { replace: true } } peers { permanent_uuid: "da0b6d111b69425585bd4008fda6771e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43521 } attrs { promote: false } }
I20240501 07:58:47.724782 2791 raft_consensus.cc:3037] T 243736c7fc354ccd86f3ac42401c61c0 P 809d92b226a94aa59287ddc81c7ff77b [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:58:47.725585 2791 raft_consensus.cc:2454] T 243736c7fc354ccd86f3ac42401c61c0 P 809d92b226a94aa59287ddc81c7ff77b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da0b6d111b69425585bd4008fda6771e in term 2.
I20240501 07:58:47.725951 5604 leader_election.cc:302] T 243736c7fc354ccd86f3ac42401c61c0 P da0b6d111b69425585bd4008fda6771e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 809d92b226a94aa59287ddc81c7ff77b, da0b6d111b69425585bd4008fda6771e; no voters:
I20240501 07:58:47.726173 5901 raft_consensus.cc:2781] T 243736c7fc354ccd86f3ac42401c61c0 P da0b6d111b69425585bd4008fda6771e [term 2 FOLLOWER]: Leader election won for term 2
I20240501 07:58:47.726349 5901 raft_consensus.cc:686] T 243736c7fc354ccd86f3ac42401c61c0 P da0b6d111b69425585bd4008fda6771e [term 2 LEADER]: Becoming Leader. State: Replica: da0b6d111b69425585bd4008fda6771e, State: Running, Role: LEADER
I20240501 07:58:47.726656 5901 consensus_queue.cc:237] T 243736c7fc354ccd86f3ac42401c61c0 P da0b6d111b69425585bd4008fda6771e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "809d92b226a94aa59287ddc81c7ff77b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33451 } attrs { replace: true } } peers { permanent_uuid: "da0b6d111b69425585bd4008fda6771e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43521 } attrs { promote: false } }
I20240501 07:58:47.728667 2644 catalog_manager.cc:5484] T 243736c7fc354ccd86f3ac42401c61c0 P da0b6d111b69425585bd4008fda6771e reported cstate change: term changed from 1 to 2, leader changed from 809d92b226a94aa59287ddc81c7ff77b (127.0.99.129) to da0b6d111b69425585bd4008fda6771e (127.0.99.136). New cstate: current_term: 2 leader_uuid: "da0b6d111b69425585bd4008fda6771e" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "809d92b226a94aa59287ddc81c7ff77b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33451 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da0b6d111b69425585bd4008fda6771e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43521 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240501 07:58:47.734825 2634 catalog_manager.cc:5014] ChangeConfig:REMOVE_PEER RPC for tablet 243736c7fc354ccd86f3ac42401c61c0 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER failed with leader da0b6d111b69425585bd4008fda6771e (127.0.99.136:43521) due to error UNKNOWN_ERROR; will retry: Illegal state: Leader has not yet committed an operation in its own term
I20240501 07:58:47.786909 5903 raft_consensus.cc:979] T 3264671dc2714e98846bc54938cf358c P 853826bd6abf488a8895392f3470a546: : Instructing follower da0b6d111b69425585bd4008fda6771e to start an election
I20240501 07:58:47.787024 5903 raft_consensus.cc:1067] T 3264671dc2714e98846bc54938cf358c P 853826bd6abf488a8895392f3470a546 [term 1 LEADER]: Signalling peer da0b6d111b69425585bd4008fda6771e to start an election
I20240501 07:58:47.787636 5661 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3264671dc2714e98846bc54938cf358c"
dest_uuid: "da0b6d111b69425585bd4008fda6771e"
from {username='slave'} at 127.0.99.130:37659
I20240501 07:58:47.787776 5661 raft_consensus.cc:482] T 3264671dc2714e98846bc54938cf358c P da0b6d111b69425585bd4008fda6771e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:58:47.787822 5661 raft_consensus.cc:3037] T 3264671dc2714e98846bc54938cf358c P da0b6d111b69425585bd4008fda6771e [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:58:47.788668 5661 raft_consensus.cc:504] T 3264671dc2714e98846bc54938cf358c P da0b6d111b69425585bd4008fda6771e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "853826bd6abf488a8895392f3470a546" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39801 } attrs { replace: true } } peers { permanent_uuid: "da0b6d111b69425585bd4008fda6771e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43521 } attrs { promote: false } }
I20240501 07:58:47.789031 5661 leader_election.cc:288] T 3264671dc2714e98846bc54938cf358c P da0b6d111b69425585bd4008fda6771e [CANDIDATE]: Term 2 election: Requested vote from peers 853826bd6abf488a8895392f3470a546 (127.0.99.130:39801)
I20240501 07:58:47.789451 2922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3264671dc2714e98846bc54938cf358c" candidate_uuid: "da0b6d111b69425585bd4008fda6771e" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "853826bd6abf488a8895392f3470a546"
I20240501 07:58:47.789569 2922 raft_consensus.cc:3032] T 3264671dc2714e98846bc54938cf358c P 853826bd6abf488a8895392f3470a546 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 07:58:47.789638 2922 raft_consensus.cc:729] T 3264671dc2714e98846bc54938cf358c P 853826bd6abf488a8895392f3470a546 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 853826bd6abf488a8895392f3470a546, State: Running, Role: LEADER
I20240501 07:58:47.789793 2922 consensus_queue.cc:260] T 3264671dc2714e98846bc54938cf358c P 853826bd6abf488a8895392f3470a546 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "853826bd6abf488a8895392f3470a546" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39801 } attrs { replace: true } } peers { permanent_uuid: "da0b6d111b69425585bd4008fda6771e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43521 } attrs { promote: false } }
I20240501 07:58:47.789930 2922 raft_consensus.cc:3037] T 3264671dc2714e98846bc54938cf358c P 853826bd6abf488a8895392f3470a546 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:58:47.790726 2922 raft_consensus.cc:2454] T 3264671dc2714e98846bc54938cf358c P 853826bd6abf488a8895392f3470a546 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da0b6d111b69425585bd4008fda6771e in term 2.
I20240501 07:58:47.791038 5604 leader_election.cc:302] T 3264671dc2714e98846bc54938cf358c P da0b6d111b69425585bd4008fda6771e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 853826bd6abf488a8895392f3470a546, da0b6d111b69425585bd4008fda6771e; no voters:
I20240501 07:58:47.791209 5901 raft_consensus.cc:2781] T 3264671dc2714e98846bc54938cf358c P da0b6d111b69425585bd4008fda6771e [term 2 FOLLOWER]: Leader election won for term 2
I20240501 07:58:47.791294 5901 raft_consensus.cc:686] T 3264671dc2714e98846bc54938cf358c P da0b6d111b69425585bd4008fda6771e [term 2 LEADER]: Becoming Leader. State: Replica: da0b6d111b69425585bd4008fda6771e, State: Running, Role: LEADER
I20240501 07:58:47.791433 5901 consensus_queue.cc:237] T 3264671dc2714e98846bc54938cf358c P da0b6d111b69425585bd4008fda6771e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "853826bd6abf488a8895392f3470a546" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39801 } attrs { replace: true } } peers { permanent_uuid: "da0b6d111b69425585bd4008fda6771e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43521 } attrs { promote: false } }
I20240501 07:58:47.792897 2644 catalog_manager.cc:5484] T 3264671dc2714e98846bc54938cf358c P da0b6d111b69425585bd4008fda6771e reported cstate change: term changed from 1 to 2, leader changed from 853826bd6abf488a8895392f3470a546 (127.0.99.130) to da0b6d111b69425585bd4008fda6771e (127.0.99.136). New cstate: current_term: 2 leader_uuid: "da0b6d111b69425585bd4008fda6771e" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "853826bd6abf488a8895392f3470a546" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39801 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da0b6d111b69425585bd4008fda6771e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43521 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 07:58:48.237424 2922 raft_consensus.cc:1261] T 3264671dc2714e98846bc54938cf358c P 853826bd6abf488a8895392f3470a546 [term 2 FOLLOWER]: Refusing update from remote peer da0b6d111b69425585bd4008fda6771e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 07:58:48.238127 5901 consensus_queue.cc:1035] T 3264671dc2714e98846bc54938cf358c P da0b6d111b69425585bd4008fda6771e [LEADER]: Connected to new peer: Peer: permanent_uuid: "853826bd6abf488a8895392f3470a546" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39801 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 07:58:48.341357 2791 raft_consensus.cc:1261] T 243736c7fc354ccd86f3ac42401c61c0 P 809d92b226a94aa59287ddc81c7ff77b [term 2 FOLLOWER]: Refusing update from remote peer da0b6d111b69425585bd4008fda6771e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 07:58:48.341915 5901 consensus_queue.cc:1035] T 243736c7fc354ccd86f3ac42401c61c0 P da0b6d111b69425585bd4008fda6771e [LEADER]: Connected to new peer: Peer: permanent_uuid: "809d92b226a94aa59287ddc81c7ff77b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 33451 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 07:58:48.363801 5661 consensus_queue.cc:237] T 243736c7fc354ccd86f3ac42401c61c0 P da0b6d111b69425585bd4008fda6771e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "da0b6d111b69425585bd4008fda6771e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43521 } attrs { promote: false } }
I20240501 07:58:48.364485 6011 raft_consensus.cc:2932] T 243736c7fc354ccd86f3ac42401c61c0 P da0b6d111b69425585bd4008fda6771e [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 809d92b226a94aa59287ddc81c7ff77b (127.0.99.129) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "da0b6d111b69425585bd4008fda6771e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43521 } attrs { promote: false } } }
I20240501 07:58:48.365883 2634 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 243736c7fc354ccd86f3ac42401c61c0 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240501 07:58:48.366870 2644 catalog_manager.cc:5484] T 243736c7fc354ccd86f3ac42401c61c0 P da0b6d111b69425585bd4008fda6771e reported cstate change: config changed from index 5 to 7, VOTER 809d92b226a94aa59287ddc81c7ff77b (127.0.99.129) evicted. New cstate: current_term: 2 leader_uuid: "da0b6d111b69425585bd4008fda6771e" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "da0b6d111b69425585bd4008fda6771e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43521 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 07:58:48.369663 2770 tablet_service.cc:1514] Processing DeleteTablet for tablet 243736c7fc354ccd86f3ac42401c61c0 with delete_type TABLET_DATA_TOMBSTONED (TS 809d92b226a94aa59287ddc81c7ff77b not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:33210
I20240501 07:58:48.369990 6016 tablet_replica.cc:330] T 243736c7fc354ccd86f3ac42401c61c0 P 809d92b226a94aa59287ddc81c7ff77b: stopping tablet replica
I20240501 07:58:48.370129 6016 raft_consensus.cc:2229] T 243736c7fc354ccd86f3ac42401c61c0 P 809d92b226a94aa59287ddc81c7ff77b [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:48.370245 6016 raft_consensus.cc:2258] T 243736c7fc354ccd86f3ac42401c61c0 P 809d92b226a94aa59287ddc81c7ff77b [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:48.371553 6016 ts_tablet_manager.cc:1875] T 243736c7fc354ccd86f3ac42401c61c0 P 809d92b226a94aa59287ddc81c7ff77b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:58:48.373389 6016 ts_tablet_manager.cc:1888] T 243736c7fc354ccd86f3ac42401c61c0 P 809d92b226a94aa59287ddc81c7ff77b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240501 07:58:48.373456 6016 log.cc:1196] T 243736c7fc354ccd86f3ac42401c61c0 P 809d92b226a94aa59287ddc81c7ff77b: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/243736c7fc354ccd86f3ac42401c61c0
I20240501 07:58:48.373997 2636 catalog_manager.cc:4830] TS 809d92b226a94aa59287ddc81c7ff77b (127.0.99.129:33451): tablet 243736c7fc354ccd86f3ac42401c61c0 (table rebalance_test_table_4 [id=c04f011aac574a36aceb005abe2af590]) successfully deleted
I20240501 07:58:48.423337 5661 consensus_queue.cc:237] T 3264671dc2714e98846bc54938cf358c P da0b6d111b69425585bd4008fda6771e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "da0b6d111b69425585bd4008fda6771e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43521 } attrs { promote: false } }
I20240501 07:58:48.424008 6011 raft_consensus.cc:2932] T 3264671dc2714e98846bc54938cf358c P da0b6d111b69425585bd4008fda6771e [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 853826bd6abf488a8895392f3470a546 (127.0.99.130) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "da0b6d111b69425585bd4008fda6771e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43521 } attrs { promote: false } } }
I20240501 07:58:48.425356 2634 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 3264671dc2714e98846bc54938cf358c with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240501 07:58:48.426296 2644 catalog_manager.cc:5484] T 3264671dc2714e98846bc54938cf358c P da0b6d111b69425585bd4008fda6771e reported cstate change: config changed from index 5 to 7, VOTER 853826bd6abf488a8895392f3470a546 (127.0.99.130) evicted. New cstate: current_term: 2 leader_uuid: "da0b6d111b69425585bd4008fda6771e" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "da0b6d111b69425585bd4008fda6771e" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43521 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 07:58:48.428840 2901 tablet_service.cc:1514] Processing DeleteTablet for tablet 3264671dc2714e98846bc54938cf358c with delete_type TABLET_DATA_TOMBSTONED (TS 853826bd6abf488a8895392f3470a546 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:37258
I20240501 07:58:48.429265 6017 tablet_replica.cc:330] T 3264671dc2714e98846bc54938cf358c P 853826bd6abf488a8895392f3470a546: stopping tablet replica
I20240501 07:58:48.429400 6017 raft_consensus.cc:2229] T 3264671dc2714e98846bc54938cf358c P 853826bd6abf488a8895392f3470a546 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 07:58:48.429507 6017 raft_consensus.cc:2258] T 3264671dc2714e98846bc54938cf358c P 853826bd6abf488a8895392f3470a546 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 07:58:48.430786 6017 ts_tablet_manager.cc:1875] T 3264671dc2714e98846bc54938cf358c P 853826bd6abf488a8895392f3470a546: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:58:48.432624 6017 ts_tablet_manager.cc:1888] T 3264671dc2714e98846bc54938cf358c P 853826bd6abf488a8895392f3470a546: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240501 07:58:48.432696 6017 log.cc:1196] T 3264671dc2714e98846bc54938cf358c P 853826bd6abf488a8895392f3470a546: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/3264671dc2714e98846bc54938cf358c
I20240501 07:58:48.433261 2636 catalog_manager.cc:4830] TS 853826bd6abf488a8895392f3470a546 (127.0.99.130:39801): tablet 3264671dc2714e98846bc54938cf358c (table rebalance_test_table_2 [id=82153012ad124c3398cc16ce35fb1b37]) successfully deleted
I20240501 07:58:48.520061 3903 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:48.521503 4437 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:48.521522 4166 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:48.528550 2770 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:48.528653 5648 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:48.530239 4034 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:48.543344 4306 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:48.569612 2901 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:48.650655 4306 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:48.651695 2770 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:48.653306 3903 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:48.659478 4166 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:48.664369 4034 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:48.666600 5648 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:48.673415 2901 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:48.693665 4437 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:48.781461 3903 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:48.782141 2770 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:48.785427 4166 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:48.798887 4437 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:48.800051 4034 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:48.806501 2901 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:48.806988 5648 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:58:48.809276 4306 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
95f14cded5744864a8da22ac2b8b06fa | 127.0.99.190:42355 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
allow_unsafe_replication_factor | true | runtime,unsafe | all 1 server(s) checked
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.0.99.148:34309 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
2471afee7ac94728a1506dc907ef8fbb | 127.0.99.134:38167 | HEALTHY | <none> | 2 | 0
2b98ac09d43149c5bcfd2c0080ea5379 | 127.0.99.133:40367 | HEALTHY | <none> | 2 | 0
809d92b226a94aa59287ddc81c7ff77b | 127.0.99.129:33451 | HEALTHY | <none> | 2 | 0
853826bd6abf488a8895392f3470a546 | 127.0.99.130:39801 | HEALTHY | <none> | 2 | 0
9846630212094f49930ef629ee885368 | 127.0.99.131:33467 | HEALTHY | <none> | 2 | 0
98526b95c3e94668a1673fe660495876 | 127.0.99.135:39073 | HEALTHY | <none> | 2 | 0
cb0a6f265d7e4362aea53c916ceb49ec | 127.0.99.132:43375 | HEALTHY | <none> | 2 | 0
da0b6d111b69425585bd4008fda6771e | 127.0.99.136:43521 | HEALTHY | <none> | 2 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 8
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 8 server(s) checked
local_ip_for_outbound_sockets | 127.0.99.129 | experimental | 127.0.99.129:33451
local_ip_for_outbound_sockets | 127.0.99.130 | experimental | 127.0.99.130:39801
local_ip_for_outbound_sockets | 127.0.99.131 | experimental | 127.0.99.131:33467
local_ip_for_outbound_sockets | 127.0.99.132 | experimental | 127.0.99.132:43375
local_ip_for_outbound_sockets | 127.0.99.133 | experimental | 127.0.99.133:40367
local_ip_for_outbound_sockets | 127.0.99.134 | experimental | 127.0.99.134:38167
local_ip_for_outbound_sockets | 127.0.99.135 | experimental | 127.0.99.135:39073
local_ip_for_outbound_sockets | 127.0.99.136 | experimental | 127.0.99.136:43521
never_fsync | true | unsafe,advanced | all 8 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 8 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 8 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 8 server(s) checked
server_dump_info_format | pb | hidden | all 8 server(s) checked
server_dump_info_path | /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.0.99.129:33451
server_dump_info_path | /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.0.99.130:39801
server_dump_info_path | /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.0.99.131:33467
server_dump_info_path | /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.0.99.132:43375
server_dump_info_path | /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.0.99.133:40367
server_dump_info_path | /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.0.99.134:38167
server_dump_info_path | /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden | 127.0.99.135:39073
server_dump_info_path | /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb | hidden | 127.0.99.136:43521
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.0.99.148:34309 | all 8 server(s) checked
time_source | builtin | all 8 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 9 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 1 | HEALTHY | 1 | 1 | 0 | 0 | 0
rebalance_test_table_0 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_4 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 2
First Quartile | 2
Median | 2
Third Quartile | 2
Maximum | 2
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 8
Tables | 6
Tablets | 16
Replicas | 16
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240501 07:58:48.843827 398 log_verifier.cc:126] Checking tablet 0b2690f2189b43f9a45e1a90605623ad
I20240501 07:58:48.850837 398 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 0b2690f2189b43f9a45e1a90605623ad
I20240501 07:58:48.850905 398 log_verifier.cc:126] Checking tablet 102891a28f95444dac4798b2aace3857
I20240501 07:58:48.856678 398 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 102891a28f95444dac4798b2aace3857
I20240501 07:58:48.856746 398 log_verifier.cc:126] Checking tablet 149f1f36b61a4d148c2003caaa7fce0c
I20240501 07:58:48.862587 398 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 149f1f36b61a4d148c2003caaa7fce0c
I20240501 07:58:48.862663 398 log_verifier.cc:126] Checking tablet 14a68b0c3f984b999ddc7856ca076c8f
I20240501 07:58:48.868292 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 14a68b0c3f984b999ddc7856ca076c8f
I20240501 07:58:48.868359 398 log_verifier.cc:126] Checking tablet 18f0be6c9b944ce1b63600f456f272dd
I20240501 07:58:48.874298 398 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 18f0be6c9b944ce1b63600f456f272dd
I20240501 07:58:48.874361 398 log_verifier.cc:126] Checking tablet 243736c7fc354ccd86f3ac42401c61c0
I20240501 07:58:48.882722 398 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 243736c7fc354ccd86f3ac42401c61c0
I20240501 07:58:48.882795 398 log_verifier.cc:126] Checking tablet 3264671dc2714e98846bc54938cf358c
I20240501 07:58:48.891325 398 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 3264671dc2714e98846bc54938cf358c
I20240501 07:58:48.891399 398 log_verifier.cc:126] Checking tablet 3e1613b40cab4477952e3c04f2b16e5d
I20240501 07:58:48.896862 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 3e1613b40cab4477952e3c04f2b16e5d
I20240501 07:58:48.896929 398 log_verifier.cc:126] Checking tablet 444c036684fe4852ba6aaab77411e3a4
I20240501 07:58:48.902437 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 444c036684fe4852ba6aaab77411e3a4
I20240501 07:58:48.902503 398 log_verifier.cc:126] Checking tablet c4d49398b37446c383dc0be576a3263f
I20240501 07:58:48.907114 398 log_verifier.cc:177] Verified matching terms for 2 ops in tablet c4d49398b37446c383dc0be576a3263f
I20240501 07:58:48.907173 398 log_verifier.cc:126] Checking tablet c63214bc45b3424492cc114662f6839e
I20240501 07:58:48.912843 398 log_verifier.cc:177] Verified matching terms for 7 ops in tablet c63214bc45b3424492cc114662f6839e
I20240501 07:58:48.912906 398 log_verifier.cc:126] Checking tablet c64dae3606e049e2a72b00148fffc748
I20240501 07:58:48.918888 398 log_verifier.cc:177] Verified matching terms for 7 ops in tablet c64dae3606e049e2a72b00148fffc748
I20240501 07:58:48.918951 398 log_verifier.cc:126] Checking tablet db9747650ebf4556ac30e4138821695b
I20240501 07:58:48.924842 398 log_verifier.cc:177] Verified matching terms for 7 ops in tablet db9747650ebf4556ac30e4138821695b
I20240501 07:58:48.924906 398 log_verifier.cc:126] Checking tablet ef28ebd14a684f1f9f5d6718bf8284b4
I20240501 07:58:48.930738 398 log_verifier.cc:177] Verified matching terms for 7 ops in tablet ef28ebd14a684f1f9f5d6718bf8284b4
I20240501 07:58:48.930799 398 log_verifier.cc:126] Checking tablet f4277f2f67dd492297a8b36edc366bba
I20240501 07:58:48.936280 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet f4277f2f67dd492297a8b36edc366bba
I20240501 07:58:48.936339 398 log_verifier.cc:126] Checking tablet fa6cf4447cfe49ecbbfe94accb24aa97
I20240501 07:58:48.942066 398 log_verifier.cc:177] Verified matching terms for 7 ops in tablet fa6cf4447cfe49ecbbfe94accb24aa97
I20240501 07:58:48.942513 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tbgEt/build/debug/bin/kudu with pid 2709
I20240501 07:58:48.949517 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tbgEt/build/debug/bin/kudu with pid 2840
I20240501 07:58:48.956488 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tbgEt/build/debug/bin/kudu with pid 3841
I20240501 07:58:48.963157 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tbgEt/build/debug/bin/kudu with pid 3972
I20240501 07:58:48.969107 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tbgEt/build/debug/bin/kudu with pid 4102
I20240501 07:58:48.975152 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tbgEt/build/debug/bin/kudu with pid 4236
I20240501 07:58:48.980860 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tbgEt/build/debug/bin/kudu with pid 4375
I20240501 07:58:48.986444 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tbgEt/build/debug/bin/kudu with pid 5586
I20240501 07:58:48.991981 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tbgEt/build/debug/bin/kudu with pid 2618
2024-05-01T07:58:48Z chronyd exiting
[ OK ] RebalanceParamTest.Rebalance/1 (16126 ms)
[----------] 1 test from RebalanceParamTest (16126 ms total)
[----------] 1 test from DDLDuringRebalancingTest
[ RUN ] DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing/1
I20240501 07:58:49.031949 398 test_util.cc:268] Using random seed: -615531261
I20240501 07:58:49.032972 398 ts_itest-base.cc:112] Starting cluster with:
I20240501 07:58:49.033025 398 ts_itest-base.cc:113] --------------
I20240501 07:58:49.033049 398 ts_itest-base.cc:114] 8 tablet servers
I20240501 07:58:49.033069 398 ts_itest-base.cc:115] 3 replicas per TS
I20240501 07:58:49.033087 398 ts_itest-base.cc:116] --------------
2024-05-01T07:58:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T07:58:49Z Disabled control of system clock
I20240501 07:58:49.041085 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tbgEt/build/debug/bin/kudu
/tmp/dist-test-task7tbgEt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.99.190:38383
--webserver_interface=127.0.99.190
--webserver_port=0
--builtin_ntp_servers=127.0.99.148:34871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.99.190:38383
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false with env {}
W20240501 07:58:49.147774 6142 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:58:49.148000 6142 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 07:58:49.148039 6142 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:58:49.151136 6142 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 07:58:49.151206 6142 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 07:58:49.151229 6142 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 07:58:49.151247 6142 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 07:58:49.151265 6142 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 07:58:49.154978 6142 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:34871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.0.99.190:38383
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.99.190:38383
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.0.99.190
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.6142
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 07:43:16 UTC on 1a15035a6830
build id 69
I20240501 07:58:49.155970 6142 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 07:58:49.157181 6142 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:58:49.164726 6148 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:58:49.165724 6147 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:58:49.166563 6150 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:58:49.167656 6142 server_base.cc:1034] running on GCE node
I20240501 07:58:49.168215 6142 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 07:58:49.169235 6142 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 07:58:49.170390 6142 hybrid_clock.cc:648] HybridClock initialized: now 1714550329170383 us; error 25 us; skew 500 ppm
I20240501 07:58:49.172305 6142 webserver.cc:460] Webserver started at http://127.0.99.190:34649/ using document root <none> and password file <none>
I20240501 07:58:49.172854 6142 fs_manager.cc:359] Metadata directory not provided
I20240501 07:58:49.172917 6142 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:58:49.173151 6142 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 07:58:49.174808 6142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "0bc706e85ce74cef97a4827ea4d4aa65"
format_stamp: "Formatted at 2024-05-01 07:58:49 on dist-test-slave-40g1"
I20240501 07:58:49.175410 6142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "0bc706e85ce74cef97a4827ea4d4aa65"
format_stamp: "Formatted at 2024-05-01 07:58:49 on dist-test-slave-40g1"
I20240501 07:58:49.179008 6142 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240501 07:58:49.181362 6156 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:49.182391 6142 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240501 07:58:49.182518 6142 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/master-0/wal
uuid: "0bc706e85ce74cef97a4827ea4d4aa65"
format_stamp: "Formatted at 2024-05-01 07:58:49 on dist-test-slave-40g1"
I20240501 07:58:49.182613 6142 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:58:49.194087 6142 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:58:49.194746 6142 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 07:58:49.194897 6142 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:58:49.201177 6142 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.190:38383
I20240501 07:58:49.201197 6208 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.190:38383 every 8 connection(s)
I20240501 07:58:49.202143 6142 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240501 07:58:49.204609 6209 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:49.206228 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tbgEt/build/debug/bin/kudu as pid 6142
I20240501 07:58:49.206321 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240501 07:58:49.209905 6209 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0bc706e85ce74cef97a4827ea4d4aa65: Bootstrap starting.
I20240501 07:58:49.211889 6209 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0bc706e85ce74cef97a4827ea4d4aa65: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:49.212677 6209 log.cc:824] T 00000000000000000000000000000000 P 0bc706e85ce74cef97a4827ea4d4aa65: Log is configured to *not* fsync() on all Append() calls
I20240501 07:58:49.214461 6209 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0bc706e85ce74cef97a4827ea4d4aa65: No bootstrap required, opened a new log
I20240501 07:58:49.217177 6209 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0bc706e85ce74cef97a4827ea4d4aa65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bc706e85ce74cef97a4827ea4d4aa65" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 38383 } }
I20240501 07:58:49.217363 6209 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0bc706e85ce74cef97a4827ea4d4aa65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:49.217397 6209 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0bc706e85ce74cef97a4827ea4d4aa65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0bc706e85ce74cef97a4827ea4d4aa65, State: Initialized, Role: FOLLOWER
I20240501 07:58:49.217754 6209 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0bc706e85ce74cef97a4827ea4d4aa65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bc706e85ce74cef97a4827ea4d4aa65" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 38383 } }
I20240501 07:58:49.217855 6209 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0bc706e85ce74cef97a4827ea4d4aa65 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 07:58:49.217892 6209 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0bc706e85ce74cef97a4827ea4d4aa65 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 07:58:49.217967 6209 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0bc706e85ce74cef97a4827ea4d4aa65 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:49.218806 6209 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0bc706e85ce74cef97a4827ea4d4aa65 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bc706e85ce74cef97a4827ea4d4aa65" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 38383 } }
I20240501 07:58:49.219106 6209 leader_election.cc:302] T 00000000000000000000000000000000 P 0bc706e85ce74cef97a4827ea4d4aa65 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0bc706e85ce74cef97a4827ea4d4aa65; no voters:
I20240501 07:58:49.219331 6209 leader_election.cc:288] T 00000000000000000000000000000000 P 0bc706e85ce74cef97a4827ea4d4aa65 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 07:58:49.219671 6214 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0bc706e85ce74cef97a4827ea4d4aa65 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:49.219874 6214 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0bc706e85ce74cef97a4827ea4d4aa65 [term 1 LEADER]: Becoming Leader. State: Replica: 0bc706e85ce74cef97a4827ea4d4aa65, State: Running, Role: LEADER
I20240501 07:58:49.220172 6214 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0bc706e85ce74cef97a4827ea4d4aa65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bc706e85ce74cef97a4827ea4d4aa65" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 38383 } }
I20240501 07:58:49.220474 6209 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0bc706e85ce74cef97a4827ea4d4aa65 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 07:58:49.221690 6216 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0bc706e85ce74cef97a4827ea4d4aa65 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0bc706e85ce74cef97a4827ea4d4aa65. Latest consensus state: current_term: 1 leader_uuid: "0bc706e85ce74cef97a4827ea4d4aa65" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bc706e85ce74cef97a4827ea4d4aa65" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 38383 } } }
I20240501 07:58:49.221849 6216 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0bc706e85ce74cef97a4827ea4d4aa65 [sys.catalog]: This master's current role is: LEADER
I20240501 07:58:49.221776 6215 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0bc706e85ce74cef97a4827ea4d4aa65 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0bc706e85ce74cef97a4827ea4d4aa65" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bc706e85ce74cef97a4827ea4d4aa65" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 38383 } } }
I20240501 07:58:49.222023 6215 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0bc706e85ce74cef97a4827ea4d4aa65 [sys.catalog]: This master's current role is: LEADER
I20240501 07:58:49.222250 6223 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 07:58:49.225260 6223 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 07:58:49.230190 6223 catalog_manager.cc:1347] Generated new cluster ID: 55c9d60c5b14495fbc2b0cda7f3f47d1
I20240501 07:58:49.230268 6223 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 07:58:49.241967 6223 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 07:58:49.242934 6223 catalog_manager.cc:1504] Loading token signing keys...
I20240501 07:58:49.248360 6223 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 0bc706e85ce74cef97a4827ea4d4aa65: Generated new TSK 0
I20240501 07:58:49.249096 6223 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 07:58:49.257874 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tbgEt/build/debug/bin/kudu
/tmp/dist-test-task7tbgEt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.129:0
--local_ip_for_outbound_sockets=127.0.99.129
--webserver_interface=127.0.99.129
--webserver_port=0
--tserver_master_addrs=127.0.99.190:38383
--builtin_ntp_servers=127.0.99.148:34871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240501 07:58:49.374614 6233 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:58:49.374881 6233 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 07:58:49.374955 6233 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:58:49.378281 6233 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 07:58:49.378366 6233 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 07:58:49.378466 6233 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.129
I20240501 07:58:49.382227 6233 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:34871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.0.99.129
--webserver_port=0
--tserver_master_addrs=127.0.99.190:38383
--never_fsync=true
--heap_profile_path=/tmp/kudu.6233
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.129
--log_dir=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 07:43:16 UTC on 1a15035a6830
build id 69
I20240501 07:58:49.383180 6233 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 07:58:49.384356 6233 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:58:49.393612 6239 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:58:49.394604 6238 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:58:49.395380 6241 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:58:49.396446 6233 server_base.cc:1034] running on GCE node
I20240501 07:58:49.396891 6233 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 07:58:49.397545 6233 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 07:58:49.398706 6233 hybrid_clock.cc:648] HybridClock initialized: now 1714550329398695 us; error 27 us; skew 500 ppm
I20240501 07:58:49.400718 6233 webserver.cc:460] Webserver started at http://127.0.99.129:35229/ using document root <none> and password file <none>
I20240501 07:58:49.401332 6233 fs_manager.cc:359] Metadata directory not provided
I20240501 07:58:49.401391 6233 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:58:49.401571 6233 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 07:58:49.403390 6233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "8b8b3ad3aa8f4053b2167dd550f2086f"
format_stamp: "Formatted at 2024-05-01 07:58:49 on dist-test-slave-40g1"
I20240501 07:58:49.403879 6233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "8b8b3ad3aa8f4053b2167dd550f2086f"
format_stamp: "Formatted at 2024-05-01 07:58:49 on dist-test-slave-40g1"
I20240501 07:58:49.407338 6233 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240501 07:58:49.409575 6247 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:49.410636 6233 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240501 07:58:49.410739 6233 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "8b8b3ad3aa8f4053b2167dd550f2086f"
format_stamp: "Formatted at 2024-05-01 07:58:49 on dist-test-slave-40g1"
I20240501 07:58:49.410849 6233 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:58:49.427450 6233 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:58:49.428077 6233 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 07:58:49.428242 6233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:58:49.428787 6233 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 07:58:49.429721 6233 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 07:58:49.429769 6233 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:49.429812 6233 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 07:58:49.429842 6233 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:49.437923 6233 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.129:46875
I20240501 07:58:49.437955 6360 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.129:46875 every 8 connection(s)
I20240501 07:58:49.438884 6233 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240501 07:58:49.444425 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tbgEt/build/debug/bin/kudu as pid 6233
I20240501 07:58:49.444533 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240501 07:58:49.446902 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tbgEt/build/debug/bin/kudu
/tmp/dist-test-task7tbgEt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.130:0
--local_ip_for_outbound_sockets=127.0.99.130
--webserver_interface=127.0.99.130
--webserver_port=0
--tserver_master_addrs=127.0.99.190:38383
--builtin_ntp_servers=127.0.99.148:34871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 07:58:49.448434 6361 heartbeater.cc:347] Connected to a master server at 127.0.99.190:38383
I20240501 07:58:49.448657 6361 heartbeater.cc:464] Registering TS with master...
I20240501 07:58:49.449268 6361 heartbeater.cc:511] Master 127.0.99.190:38383 requested a full tablet report, sending...
I20240501 07:58:49.450776 6173 ts_manager.cc:194] Registered new tserver with Master: 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129:46875)
I20240501 07:58:49.451984 6173 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.129:59221
W20240501 07:58:49.557298 6364 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:58:49.557574 6364 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 07:58:49.557655 6364 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:58:49.560779 6364 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 07:58:49.560869 6364 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 07:58:49.561010 6364 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.130
I20240501 07:58:49.564682 6364 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:34871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.0.99.130
--webserver_port=0
--tserver_master_addrs=127.0.99.190:38383
--never_fsync=true
--heap_profile_path=/tmp/kudu.6364
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.130
--log_dir=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 07:43:16 UTC on 1a15035a6830
build id 69
I20240501 07:58:49.565681 6364 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 07:58:49.566839 6364 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:58:49.575690 6369 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:58:49.576460 6370 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:58:49.577538 6372 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:58:49.578456 6364 server_base.cc:1034] running on GCE node
I20240501 07:58:49.578918 6364 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 07:58:49.579550 6364 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 07:58:49.580703 6364 hybrid_clock.cc:648] HybridClock initialized: now 1714550329580674 us; error 48 us; skew 500 ppm
I20240501 07:58:49.582823 6364 webserver.cc:460] Webserver started at http://127.0.99.130:39501/ using document root <none> and password file <none>
I20240501 07:58:49.583400 6364 fs_manager.cc:359] Metadata directory not provided
I20240501 07:58:49.583456 6364 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:58:49.583673 6364 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 07:58:49.585419 6364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "8315d22c56f7439a9e1fc82f153f018c"
format_stamp: "Formatted at 2024-05-01 07:58:49 on dist-test-slave-40g1"
I20240501 07:58:49.585883 6364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "8315d22c56f7439a9e1fc82f153f018c"
format_stamp: "Formatted at 2024-05-01 07:58:49 on dist-test-slave-40g1"
I20240501 07:58:49.589275 6364 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240501 07:58:49.591699 6378 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:49.592860 6364 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 07:58:49.593026 6364 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "8315d22c56f7439a9e1fc82f153f018c"
format_stamp: "Formatted at 2024-05-01 07:58:49 on dist-test-slave-40g1"
I20240501 07:58:49.593125 6364 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:58:49.609761 6364 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:58:49.610421 6364 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 07:58:49.610594 6364 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:58:49.611140 6364 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 07:58:49.612056 6364 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 07:58:49.612105 6364 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:49.612166 6364 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 07:58:49.612196 6364 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:49.620661 6364 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.130:32941
I20240501 07:58:49.620720 6491 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.130:32941 every 8 connection(s)
I20240501 07:58:49.621711 6364 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240501 07:58:49.623169 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tbgEt/build/debug/bin/kudu as pid 6364
I20240501 07:58:49.623263 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240501 07:58:49.625494 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tbgEt/build/debug/bin/kudu
/tmp/dist-test-task7tbgEt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.131:0
--local_ip_for_outbound_sockets=127.0.99.131
--webserver_interface=127.0.99.131
--webserver_port=0
--tserver_master_addrs=127.0.99.190:38383
--builtin_ntp_servers=127.0.99.148:34871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 07:58:49.632194 6492 heartbeater.cc:347] Connected to a master server at 127.0.99.190:38383
I20240501 07:58:49.632457 6492 heartbeater.cc:464] Registering TS with master...
I20240501 07:58:49.633149 6492 heartbeater.cc:511] Master 127.0.99.190:38383 requested a full tablet report, sending...
I20240501 07:58:49.634181 6173 ts_manager.cc:194] Registered new tserver with Master: 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130:32941)
I20240501 07:58:49.634748 6173 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.130:50859
W20240501 07:58:49.737164 6495 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:58:49.737422 6495 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 07:58:49.737497 6495 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:58:49.740689 6495 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 07:58:49.740772 6495 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 07:58:49.740859 6495 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.131
I20240501 07:58:49.744637 6495 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:34871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.0.99.131
--webserver_port=0
--tserver_master_addrs=127.0.99.190:38383
--never_fsync=true
--heap_profile_path=/tmp/kudu.6495
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.131
--log_dir=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 07:43:16 UTC on 1a15035a6830
build id 69
I20240501 07:58:49.745645 6495 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 07:58:49.746825 6495 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:58:49.756103 6503 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:58:49.756795 6500 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:58:49.757467 6501 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:58:49.758585 6495 server_base.cc:1034] running on GCE node
I20240501 07:58:49.759122 6495 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 07:58:49.759821 6495 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 07:58:49.761019 6495 hybrid_clock.cc:648] HybridClock initialized: now 1714550329761005 us; error 36 us; skew 500 ppm
I20240501 07:58:49.763258 6495 webserver.cc:460] Webserver started at http://127.0.99.131:43619/ using document root <none> and password file <none>
I20240501 07:58:49.763849 6495 fs_manager.cc:359] Metadata directory not provided
I20240501 07:58:49.763913 6495 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:58:49.764098 6495 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 07:58:49.765858 6495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9"
format_stamp: "Formatted at 2024-05-01 07:58:49 on dist-test-slave-40g1"
I20240501 07:58:49.766343 6495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9"
format_stamp: "Formatted at 2024-05-01 07:58:49 on dist-test-slave-40g1"
I20240501 07:58:49.769878 6495 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.002s
I20240501 07:58:49.772413 6509 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:49.773557 6495 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240501 07:58:49.773675 6495 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9"
format_stamp: "Formatted at 2024-05-01 07:58:49 on dist-test-slave-40g1"
I20240501 07:58:49.773774 6495 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:58:49.806607 6495 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:58:49.807297 6495 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 07:58:49.807476 6495 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:58:49.808012 6495 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 07:58:49.808923 6495 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 07:58:49.808996 6495 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:49.809067 6495 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 07:58:49.809093 6495 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:49.817755 6495 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.131:34471
I20240501 07:58:49.817775 6622 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.131:34471 every 8 connection(s)
I20240501 07:58:49.818785 6495 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240501 07:58:49.821903 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tbgEt/build/debug/bin/kudu as pid 6495
I20240501 07:58:49.822034 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240501 07:58:49.824525 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tbgEt/build/debug/bin/kudu
/tmp/dist-test-task7tbgEt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.132:0
--local_ip_for_outbound_sockets=127.0.99.132
--webserver_interface=127.0.99.132
--webserver_port=0
--tserver_master_addrs=127.0.99.190:38383
--builtin_ntp_servers=127.0.99.148:34871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 07:58:49.829245 6623 heartbeater.cc:347] Connected to a master server at 127.0.99.190:38383
I20240501 07:58:49.829491 6623 heartbeater.cc:464] Registering TS with master...
I20240501 07:58:49.830076 6623 heartbeater.cc:511] Master 127.0.99.190:38383 requested a full tablet report, sending...
I20240501 07:58:49.831161 6173 ts_manager.cc:194] Registered new tserver with Master: 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131:34471)
I20240501 07:58:49.831738 6173 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.131:50853
W20240501 07:58:49.937448 6626 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:58:49.937687 6626 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 07:58:49.937741 6626 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:58:49.940845 6626 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 07:58:49.940915 6626 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 07:58:49.941037 6626 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.132
I20240501 07:58:49.944706 6626 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:34871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.0.99.132
--webserver_port=0
--tserver_master_addrs=127.0.99.190:38383
--never_fsync=true
--heap_profile_path=/tmp/kudu.6626
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.132
--log_dir=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 07:43:16 UTC on 1a15035a6830
build id 69
I20240501 07:58:49.945693 6626 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 07:58:49.946830 6626 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:58:49.955749 6634 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:58:49.956665 6631 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:58:49.957641 6632 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:58:49.958612 6626 server_base.cc:1034] running on GCE node
I20240501 07:58:49.959020 6626 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 07:58:49.959642 6626 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 07:58:49.960830 6626 hybrid_clock.cc:648] HybridClock initialized: now 1714550329960806 us; error 40 us; skew 500 ppm
I20240501 07:58:49.962935 6626 webserver.cc:460] Webserver started at http://127.0.99.132:46487/ using document root <none> and password file <none>
I20240501 07:58:49.963527 6626 fs_manager.cc:359] Metadata directory not provided
I20240501 07:58:49.963587 6626 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:58:49.963761 6626 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 07:58:49.965569 6626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "1f9cb72be98941b5a8e504cbe34132b9"
format_stamp: "Formatted at 2024-05-01 07:58:49 on dist-test-slave-40g1"
I20240501 07:58:49.966074 6626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "1f9cb72be98941b5a8e504cbe34132b9"
format_stamp: "Formatted at 2024-05-01 07:58:49 on dist-test-slave-40g1"
I20240501 07:58:49.969738 6626 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240501 07:58:49.972276 6640 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:49.973527 6626 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240501 07:58:49.973637 6626 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "1f9cb72be98941b5a8e504cbe34132b9"
format_stamp: "Formatted at 2024-05-01 07:58:49 on dist-test-slave-40g1"
I20240501 07:58:49.973726 6626 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:58:49.988735 6626 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:58:49.989435 6626 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 07:58:49.989585 6626 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:58:49.990123 6626 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 07:58:49.991075 6626 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 07:58:49.991122 6626 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:49.991164 6626 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 07:58:49.991186 6626 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:50.000077 6626 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.132:36773
I20240501 07:58:50.000103 6753 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.132:36773 every 8 connection(s)
I20240501 07:58:50.001138 6626 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240501 07:58:50.001472 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tbgEt/build/debug/bin/kudu as pid 6626
I20240501 07:58:50.001596 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240501 07:58:50.004007 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tbgEt/build/debug/bin/kudu
/tmp/dist-test-task7tbgEt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.133:0
--local_ip_for_outbound_sockets=127.0.99.133
--webserver_interface=127.0.99.133
--webserver_port=0
--tserver_master_addrs=127.0.99.190:38383
--builtin_ntp_servers=127.0.99.148:34871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 07:58:50.011889 6754 heartbeater.cc:347] Connected to a master server at 127.0.99.190:38383
I20240501 07:58:50.012183 6754 heartbeater.cc:464] Registering TS with master...
I20240501 07:58:50.012931 6754 heartbeater.cc:511] Master 127.0.99.190:38383 requested a full tablet report, sending...
I20240501 07:58:50.014214 6173 ts_manager.cc:194] Registered new tserver with Master: 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132:36773)
I20240501 07:58:50.014978 6173 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.132:46797
W20240501 07:58:50.119477 6757 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:58:50.119776 6757 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 07:58:50.119828 6757 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:58:50.123109 6757 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 07:58:50.123180 6757 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 07:58:50.123250 6757 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.133
I20240501 07:58:50.126899 6757 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:34871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.133:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.0.99.133
--webserver_port=0
--tserver_master_addrs=127.0.99.190:38383
--never_fsync=true
--heap_profile_path=/tmp/kudu.6757
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.133
--log_dir=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 07:43:16 UTC on 1a15035a6830
build id 69
I20240501 07:58:50.127846 6757 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 07:58:50.129027 6757 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:58:50.138378 6763 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:58:50.138928 6762 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:58:50.139799 6765 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:58:50.140836 6757 server_base.cc:1034] running on GCE node
I20240501 07:58:50.141363 6757 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 07:58:50.141980 6757 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 07:58:50.143152 6757 hybrid_clock.cc:648] HybridClock initialized: now 1714550330143118 us; error 50 us; skew 500 ppm
I20240501 07:58:50.145416 6757 webserver.cc:460] Webserver started at http://127.0.99.133:37019/ using document root <none> and password file <none>
I20240501 07:58:50.146013 6757 fs_manager.cc:359] Metadata directory not provided
I20240501 07:58:50.146068 6757 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:58:50.146225 6757 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 07:58:50.147899 6757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "74077b61d6f54342bf9736b4a9f2eade"
format_stamp: "Formatted at 2024-05-01 07:58:50 on dist-test-slave-40g1"
I20240501 07:58:50.148361 6757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "74077b61d6f54342bf9736b4a9f2eade"
format_stamp: "Formatted at 2024-05-01 07:58:50 on dist-test-slave-40g1"
I20240501 07:58:50.151890 6757 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20240501 07:58:50.154356 6771 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:50.155395 6757 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240501 07:58:50.155494 6757 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "74077b61d6f54342bf9736b4a9f2eade"
format_stamp: "Formatted at 2024-05-01 07:58:50 on dist-test-slave-40g1"
I20240501 07:58:50.155589 6757 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:58:50.183923 6757 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:58:50.184614 6757 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 07:58:50.184784 6757 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:58:50.185406 6757 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 07:58:50.186684 6757 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 07:58:50.186733 6757 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:50.186782 6757 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 07:58:50.186812 6757 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:50.195782 6757 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.133:39859
I20240501 07:58:50.195813 6884 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.133:39859 every 8 connection(s)
I20240501 07:58:50.196897 6757 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 07:58:50.201189 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tbgEt/build/debug/bin/kudu as pid 6757
I20240501 07:58:50.201313 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240501 07:58:50.203791 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tbgEt/build/debug/bin/kudu
/tmp/dist-test-task7tbgEt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.134:0
--local_ip_for_outbound_sockets=127.0.99.134
--webserver_interface=127.0.99.134
--webserver_port=0
--tserver_master_addrs=127.0.99.190:38383
--builtin_ntp_servers=127.0.99.148:34871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 07:58:50.207392 6885 heartbeater.cc:347] Connected to a master server at 127.0.99.190:38383
I20240501 07:58:50.207649 6885 heartbeater.cc:464] Registering TS with master...
I20240501 07:58:50.208294 6885 heartbeater.cc:511] Master 127.0.99.190:38383 requested a full tablet report, sending...
I20240501 07:58:50.209501 6173 ts_manager.cc:194] Registered new tserver with Master: 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133:39859)
I20240501 07:58:50.210340 6173 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.133:33465
W20240501 07:58:50.315986 6888 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:58:50.316248 6888 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 07:58:50.316295 6888 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:58:50.319442 6888 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 07:58:50.319516 6888 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 07:58:50.319586 6888 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.134
I20240501 07:58:50.323515 6888 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:34871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.134:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.0.99.134
--webserver_port=0
--tserver_master_addrs=127.0.99.190:38383
--never_fsync=true
--heap_profile_path=/tmp/kudu.6888
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.134
--log_dir=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 07:43:16 UTC on 1a15035a6830
build id 69
I20240501 07:58:50.324714 6888 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 07:58:50.326246 6888 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:58:50.336212 6896 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:58:50.337352 6893 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:58:50.338032 6894 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:58:50.339098 6888 server_base.cc:1034] running on GCE node
I20240501 07:58:50.339555 6888 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 07:58:50.340190 6888 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 07:58:50.341341 6888 hybrid_clock.cc:648] HybridClock initialized: now 1714550330341319 us; error 42 us; skew 500 ppm
I20240501 07:58:50.343448 6888 webserver.cc:460] Webserver started at http://127.0.99.134:34395/ using document root <none> and password file <none>
I20240501 07:58:50.344048 6888 fs_manager.cc:359] Metadata directory not provided
I20240501 07:58:50.344110 6888 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:58:50.344295 6888 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 07:58:50.346099 6888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "13d05ebdca8c4f489c597ce550dd773d"
format_stamp: "Formatted at 2024-05-01 07:58:50 on dist-test-slave-40g1"
I20240501 07:58:50.346588 6888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "13d05ebdca8c4f489c597ce550dd773d"
format_stamp: "Formatted at 2024-05-01 07:58:50 on dist-test-slave-40g1"
I20240501 07:58:50.350106 6888 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240501 07:58:50.352499 6902 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:50.353657 6888 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240501 07:58:50.353766 6888 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "13d05ebdca8c4f489c597ce550dd773d"
format_stamp: "Formatted at 2024-05-01 07:58:50 on dist-test-slave-40g1"
I20240501 07:58:50.353873 6888 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:58:50.371758 6888 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:58:50.372427 6888 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 07:58:50.372598 6888 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:58:50.373250 6888 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 07:58:50.374238 6888 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 07:58:50.374289 6888 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:50.374352 6888 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 07:58:50.374379 6888 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:50.383216 6888 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.134:40141
I20240501 07:58:50.383247 7015 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.134:40141 every 8 connection(s)
I20240501 07:58:50.384232 6888 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240501 07:58:50.390692 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tbgEt/build/debug/bin/kudu as pid 6888
I20240501 07:58:50.390838 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240501 07:58:50.395398 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tbgEt/build/debug/bin/kudu
/tmp/dist-test-task7tbgEt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.135:0
--local_ip_for_outbound_sockets=127.0.99.135
--webserver_interface=127.0.99.135
--webserver_port=0
--tserver_master_addrs=127.0.99.190:38383
--builtin_ntp_servers=127.0.99.148:34871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 07:58:50.395627 7016 heartbeater.cc:347] Connected to a master server at 127.0.99.190:38383
I20240501 07:58:50.395864 7016 heartbeater.cc:464] Registering TS with master...
I20240501 07:58:50.396509 7016 heartbeater.cc:511] Master 127.0.99.190:38383 requested a full tablet report, sending...
I20240501 07:58:50.398072 6173 ts_manager.cc:194] Registered new tserver with Master: 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134:40141)
I20240501 07:58:50.398605 6173 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.134:35489
I20240501 07:58:50.454356 6361 heartbeater.cc:503] Master 127.0.99.190:38383 was elected leader, sending a full tablet report...
W20240501 07:58:50.509248 7019 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:58:50.509510 7019 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 07:58:50.509586 7019 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:58:50.512717 7019 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 07:58:50.512802 7019 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 07:58:50.512909 7019 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.135
I20240501 07:58:50.516856 7019 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:34871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.135:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.0.99.135
--webserver_port=0
--tserver_master_addrs=127.0.99.190:38383
--never_fsync=true
--heap_profile_path=/tmp/kudu.7019
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.135
--log_dir=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 07:43:16 UTC on 1a15035a6830
build id 69
I20240501 07:58:50.517895 7019 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 07:58:50.519079 7019 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:58:50.528472 7025 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:58:50.529376 7024 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:58:50.530299 7027 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:58:50.531437 7019 server_base.cc:1034] running on GCE node
I20240501 07:58:50.531965 7019 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 07:58:50.532634 7019 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 07:58:50.533808 7019 hybrid_clock.cc:648] HybridClock initialized: now 1714550330533756 us; error 69 us; skew 500 ppm
I20240501 07:58:50.536026 7019 webserver.cc:460] Webserver started at http://127.0.99.135:45825/ using document root <none> and password file <none>
I20240501 07:58:50.536633 7019 fs_manager.cc:359] Metadata directory not provided
I20240501 07:58:50.536689 7019 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:58:50.536859 7019 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 07:58:50.538659 7019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "a505e7cb133c4392bfda8a4a20874ae9"
format_stamp: "Formatted at 2024-05-01 07:58:50 on dist-test-slave-40g1"
I20240501 07:58:50.539124 7019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "a505e7cb133c4392bfda8a4a20874ae9"
format_stamp: "Formatted at 2024-05-01 07:58:50 on dist-test-slave-40g1"
I20240501 07:58:50.542726 7019 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20240501 07:58:50.545166 7033 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:50.546295 7019 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240501 07:58:50.546415 7019 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "a505e7cb133c4392bfda8a4a20874ae9"
format_stamp: "Formatted at 2024-05-01 07:58:50 on dist-test-slave-40g1"
I20240501 07:58:50.546504 7019 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:58:50.560017 7019 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:58:50.560648 7019 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 07:58:50.560796 7019 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:58:50.561524 7019 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 07:58:50.562587 7019 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 07:58:50.562635 7019 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:50.562681 7019 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 07:58:50.562716 7019 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:50.571141 7019 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.135:42607
I20240501 07:58:50.571322 7146 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.135:42607 every 8 connection(s)
I20240501 07:58:50.572296 7019 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240501 07:58:50.581468 7147 heartbeater.cc:347] Connected to a master server at 127.0.99.190:38383
I20240501 07:58:50.581725 7147 heartbeater.cc:464] Registering TS with master...
I20240501 07:58:50.582113 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tbgEt/build/debug/bin/kudu as pid 7019
I20240501 07:58:50.582232 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240501 07:58:50.582535 7147 heartbeater.cc:511] Master 127.0.99.190:38383 requested a full tablet report, sending...
I20240501 07:58:50.583621 6172 ts_manager.cc:194] Registered new tserver with Master: a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607)
I20240501 07:58:50.584400 6172 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.135:33295
I20240501 07:58:50.584563 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tbgEt/build/debug/bin/kudu
/tmp/dist-test-task7tbgEt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.136:0
--local_ip_for_outbound_sockets=127.0.99.136
--webserver_interface=127.0.99.136
--webserver_port=0
--tserver_master_addrs=127.0.99.190:38383
--builtin_ntp_servers=127.0.99.148:34871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 07:58:50.636670 6492 heartbeater.cc:503] Master 127.0.99.190:38383 was elected leader, sending a full tablet report...
W20240501 07:58:50.715431 7150 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:58:50.715708 7150 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 07:58:50.715772 7150 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:58:50.719066 7150 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 07:58:50.719166 7150 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 07:58:50.719270 7150 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.136
I20240501 07:58:50.723271 7150 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:34871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.136:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.0.99.136
--webserver_port=0
--tserver_master_addrs=127.0.99.190:38383
--never_fsync=true
--heap_profile_path=/tmp/kudu.7150
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.136
--log_dir=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 07:43:16 UTC on 1a15035a6830
build id 69
I20240501 07:58:50.724300 7150 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 07:58:50.725544 7150 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:58:50.734915 7155 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:58:50.735376 7156 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:58:50.736436 7158 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:58:50.737535 7150 server_base.cc:1034] running on GCE node
I20240501 07:58:50.738010 7150 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 07:58:50.738629 7150 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 07:58:50.739812 7150 hybrid_clock.cc:648] HybridClock initialized: now 1714550330739785 us; error 48 us; skew 500 ppm
I20240501 07:58:50.742070 7150 webserver.cc:460] Webserver started at http://127.0.99.136:41673/ using document root <none> and password file <none>
I20240501 07:58:50.742664 7150 fs_manager.cc:359] Metadata directory not provided
I20240501 07:58:50.742727 7150 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:58:50.742935 7150 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 07:58:50.744700 7150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "83f20e84af4c40d09a4804b6b26e697f"
format_stamp: "Formatted at 2024-05-01 07:58:50 on dist-test-slave-40g1"
I20240501 07:58:50.745251 7150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "83f20e84af4c40d09a4804b6b26e697f"
format_stamp: "Formatted at 2024-05-01 07:58:50 on dist-test-slave-40g1"
I20240501 07:58:50.748745 7150 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20240501 07:58:50.751356 7164 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:50.752501 7150 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 07:58:50.752637 7150 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "83f20e84af4c40d09a4804b6b26e697f"
format_stamp: "Formatted at 2024-05-01 07:58:50 on dist-test-slave-40g1"
I20240501 07:58:50.752735 7150 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:58:50.767939 7150 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:58:50.768572 7150 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 07:58:50.768718 7150 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:58:50.769281 7150 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 07:58:50.770148 7150 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 07:58:50.770200 7150 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:50.770241 7150 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 07:58:50.770262 7150 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 07:58:50.778960 7150 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.136:46149
I20240501 07:58:50.778946 7277 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.136:46149 every 8 connection(s)
I20240501 07:58:50.779978 7150 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240501 07:58:50.780938 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tbgEt/build/debug/bin/kudu as pid 7150
I20240501 07:58:50.781080 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240501 07:58:50.789664 7278 heartbeater.cc:347] Connected to a master server at 127.0.99.190:38383
I20240501 07:58:50.789919 7278 heartbeater.cc:464] Registering TS with master...
I20240501 07:58:50.790505 7278 heartbeater.cc:511] Master 127.0.99.190:38383 requested a full tablet report, sending...
I20240501 07:58:50.791522 6172 ts_manager.cc:194] Registered new tserver with Master: 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149)
I20240501 07:58:50.792104 6172 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.136:35431
I20240501 07:58:50.795641 398 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240501 07:58:50.805141 398 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240501 07:58:50.805233 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tbgEt/build/debug/bin/kudu with pid 6757
I20240501 07:58:50.810973 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tbgEt/build/debug/bin/kudu with pid 6888
I20240501 07:58:50.818727 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tbgEt/build/debug/bin/kudu with pid 7019
I20240501 07:58:50.826104 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tbgEt/build/debug/bin/kudu with pid 7150
I20240501 07:58:50.833994 6623 heartbeater.cc:503] Master 127.0.99.190:38383 was elected leader, sending a full tablet report...
I20240501 07:58:51.017680 6754 heartbeater.cc:503] Master 127.0.99.190:38383 was elected leader, sending a full tablet report...
I20240501 07:58:54.582995 6172 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55950:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 07:58:54.603716 6424 tablet_service.cc:1467] Processing CreateTablet for tablet e04fe36b8efa4f9584e093279f7addf4 (DEFAULT_TABLE table=rebalance_test_table_0 [id=6448eafcd04b41cba44dc175507acfc0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:54.603719 6557 tablet_service.cc:1467] Processing CreateTablet for tablet def178155f14416391e89ba972f4d723 (DEFAULT_TABLE table=rebalance_test_table_0 [id=6448eafcd04b41cba44dc175507acfc0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:54.604178 6294 tablet_service.cc:1467] Processing CreateTablet for tablet e04fe36b8efa4f9584e093279f7addf4 (DEFAULT_TABLE table=rebalance_test_table_0 [id=6448eafcd04b41cba44dc175507acfc0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:54.604184 6295 tablet_service.cc:1467] Processing CreateTablet for tablet 8480c7b51a914f03bcad7701d75e4dd4 (DEFAULT_TABLE table=rebalance_test_table_0 [id=6448eafcd04b41cba44dc175507acfc0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:54.604702 6424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e04fe36b8efa4f9584e093279f7addf4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:54.604702 6557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet def178155f14416391e89ba972f4d723. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:54.604979 6294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e04fe36b8efa4f9584e093279f7addf4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:54.605412 6556 tablet_service.cc:1467] Processing CreateTablet for tablet 8480c7b51a914f03bcad7701d75e4dd4 (DEFAULT_TABLE table=rebalance_test_table_0 [id=6448eafcd04b41cba44dc175507acfc0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:54.605666 6556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8480c7b51a914f03bcad7701d75e4dd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:54.606329 6295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8480c7b51a914f03bcad7701d75e4dd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:54.609133 6425 tablet_service.cc:1467] Processing CreateTablet for tablet 8480c7b51a914f03bcad7701d75e4dd4 (DEFAULT_TABLE table=rebalance_test_table_0 [id=6448eafcd04b41cba44dc175507acfc0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:54.609401 6425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8480c7b51a914f03bcad7701d75e4dd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:54.610482 7298 tablet_bootstrap.cc:492] T 8480c7b51a914f03bcad7701d75e4dd4 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Bootstrap starting.
I20240501 07:58:54.603716 6426 tablet_service.cc:1467] Processing CreateTablet for tablet def178155f14416391e89ba972f4d723 (DEFAULT_TABLE table=rebalance_test_table_0 [id=6448eafcd04b41cba44dc175507acfc0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:54.611717 6426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet def178155f14416391e89ba972f4d723. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:54.612618 7298 tablet_bootstrap.cc:654] T 8480c7b51a914f03bcad7701d75e4dd4 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:54.613498 7298 log.cc:824] T 8480c7b51a914f03bcad7701d75e4dd4 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Log is configured to *not* fsync() on all Append() calls
I20240501 07:58:54.613765 6688 tablet_service.cc:1467] Processing CreateTablet for tablet def178155f14416391e89ba972f4d723 (DEFAULT_TABLE table=rebalance_test_table_0 [id=6448eafcd04b41cba44dc175507acfc0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:54.614341 6687 tablet_service.cc:1467] Processing CreateTablet for tablet e04fe36b8efa4f9584e093279f7addf4 (DEFAULT_TABLE table=rebalance_test_table_0 [id=6448eafcd04b41cba44dc175507acfc0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:54.614521 7299 tablet_bootstrap.cc:492] T e04fe36b8efa4f9584e093279f7addf4 P 8315d22c56f7439a9e1fc82f153f018c: Bootstrap starting.
I20240501 07:58:54.614686 6688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet def178155f14416391e89ba972f4d723. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:54.615028 6687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e04fe36b8efa4f9584e093279f7addf4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:54.615224 7301 tablet_bootstrap.cc:492] T e04fe36b8efa4f9584e093279f7addf4 P 8b8b3ad3aa8f4053b2167dd550f2086f: Bootstrap starting.
I20240501 07:58:54.617122 7301 tablet_bootstrap.cc:654] T e04fe36b8efa4f9584e093279f7addf4 P 8b8b3ad3aa8f4053b2167dd550f2086f: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:54.617122 7299 tablet_bootstrap.cc:654] T e04fe36b8efa4f9584e093279f7addf4 P 8315d22c56f7439a9e1fc82f153f018c: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:54.617899 7299 log.cc:824] T e04fe36b8efa4f9584e093279f7addf4 P 8315d22c56f7439a9e1fc82f153f018c: Log is configured to *not* fsync() on all Append() calls
I20240501 07:58:54.617899 7301 log.cc:824] T e04fe36b8efa4f9584e093279f7addf4 P 8b8b3ad3aa8f4053b2167dd550f2086f: Log is configured to *not* fsync() on all Append() calls
I20240501 07:58:54.619580 7299 tablet_bootstrap.cc:492] T e04fe36b8efa4f9584e093279f7addf4 P 8315d22c56f7439a9e1fc82f153f018c: No bootstrap required, opened a new log
I20240501 07:58:54.619704 7298 tablet_bootstrap.cc:492] T 8480c7b51a914f03bcad7701d75e4dd4 P 3b164cbbc00f4e8eb3254ddd349a3ff9: No bootstrap required, opened a new log
I20240501 07:58:54.619746 7299 ts_tablet_manager.cc:1367] T e04fe36b8efa4f9584e093279f7addf4 P 8315d22c56f7439a9e1fc82f153f018c: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20240501 07:58:54.619839 7298 ts_tablet_manager.cc:1367] T 8480c7b51a914f03bcad7701d75e4dd4 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20240501 07:58:54.621968 7301 tablet_bootstrap.cc:492] T e04fe36b8efa4f9584e093279f7addf4 P 8b8b3ad3aa8f4053b2167dd550f2086f: No bootstrap required, opened a new log
I20240501 07:58:54.622119 7301 ts_tablet_manager.cc:1367] T e04fe36b8efa4f9584e093279f7addf4 P 8b8b3ad3aa8f4053b2167dd550f2086f: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20240501 07:58:54.624459 7298 raft_consensus.cc:348] T 8480c7b51a914f03bcad7701d75e4dd4 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } }
I20240501 07:58:54.625628 7298 raft_consensus.cc:374] T 8480c7b51a914f03bcad7701d75e4dd4 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:54.625579 7301 raft_consensus.cc:348] T e04fe36b8efa4f9584e093279f7addf4 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } }
I20240501 07:58:54.625734 7298 raft_consensus.cc:729] T 8480c7b51a914f03bcad7701d75e4dd4 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b164cbbc00f4e8eb3254ddd349a3ff9, State: Initialized, Role: FOLLOWER
I20240501 07:58:54.625876 7301 raft_consensus.cc:374] T e04fe36b8efa4f9584e093279f7addf4 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:54.625988 7301 raft_consensus.cc:729] T e04fe36b8efa4f9584e093279f7addf4 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b8b3ad3aa8f4053b2167dd550f2086f, State: Initialized, Role: FOLLOWER
I20240501 07:58:54.626255 7298 consensus_queue.cc:260] T 8480c7b51a914f03bcad7701d75e4dd4 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } }
I20240501 07:58:54.626513 7301 consensus_queue.cc:260] T e04fe36b8efa4f9584e093279f7addf4 P 8b8b3ad3aa8f4053b2167dd550f2086f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } }
I20240501 07:58:54.627079 7299 raft_consensus.cc:348] T e04fe36b8efa4f9584e093279f7addf4 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } }
I20240501 07:58:54.627269 7299 raft_consensus.cc:374] T e04fe36b8efa4f9584e093279f7addf4 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:54.627293 7298 ts_tablet_manager.cc:1398] T 8480c7b51a914f03bcad7701d75e4dd4 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20240501 07:58:54.627339 7299 raft_consensus.cc:729] T e04fe36b8efa4f9584e093279f7addf4 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8315d22c56f7439a9e1fc82f153f018c, State: Initialized, Role: FOLLOWER
I20240501 07:58:54.627465 7301 ts_tablet_manager.cc:1398] T e04fe36b8efa4f9584e093279f7addf4 P 8b8b3ad3aa8f4053b2167dd550f2086f: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20240501 07:58:54.627722 7299 consensus_queue.cc:260] T e04fe36b8efa4f9584e093279f7addf4 P 8315d22c56f7439a9e1fc82f153f018c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } }
I20240501 07:58:54.628600 7301 tablet_bootstrap.cc:492] T 8480c7b51a914f03bcad7701d75e4dd4 P 8b8b3ad3aa8f4053b2167dd550f2086f: Bootstrap starting.
I20240501 07:58:54.629150 7304 tablet_bootstrap.cc:492] T def178155f14416391e89ba972f4d723 P 1f9cb72be98941b5a8e504cbe34132b9: Bootstrap starting.
I20240501 07:58:54.629281 7299 ts_tablet_manager.cc:1398] T e04fe36b8efa4f9584e093279f7addf4 P 8315d22c56f7439a9e1fc82f153f018c: Time spent starting tablet: real 0.009s user 0.002s sys 0.003s
I20240501 07:58:54.629539 7299 tablet_bootstrap.cc:492] T 8480c7b51a914f03bcad7701d75e4dd4 P 8315d22c56f7439a9e1fc82f153f018c: Bootstrap starting.
I20240501 07:58:54.629781 7298 tablet_bootstrap.cc:492] T def178155f14416391e89ba972f4d723 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Bootstrap starting.
I20240501 07:58:54.630574 7299 tablet_bootstrap.cc:654] T 8480c7b51a914f03bcad7701d75e4dd4 P 8315d22c56f7439a9e1fc82f153f018c: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:54.632229 7301 tablet_bootstrap.cc:654] T 8480c7b51a914f03bcad7701d75e4dd4 P 8b8b3ad3aa8f4053b2167dd550f2086f: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:54.632279 7298 tablet_bootstrap.cc:654] T def178155f14416391e89ba972f4d723 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:54.632462 7304 tablet_bootstrap.cc:654] T def178155f14416391e89ba972f4d723 P 1f9cb72be98941b5a8e504cbe34132b9: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:54.633374 7304 log.cc:824] T def178155f14416391e89ba972f4d723 P 1f9cb72be98941b5a8e504cbe34132b9: Log is configured to *not* fsync() on all Append() calls
I20240501 07:58:54.634101 7299 tablet_bootstrap.cc:492] T 8480c7b51a914f03bcad7701d75e4dd4 P 8315d22c56f7439a9e1fc82f153f018c: No bootstrap required, opened a new log
I20240501 07:58:54.634102 7298 tablet_bootstrap.cc:492] T def178155f14416391e89ba972f4d723 P 3b164cbbc00f4e8eb3254ddd349a3ff9: No bootstrap required, opened a new log
I20240501 07:58:54.634187 7298 ts_tablet_manager.cc:1367] T def178155f14416391e89ba972f4d723 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 07:58:54.634239 7301 tablet_bootstrap.cc:492] T 8480c7b51a914f03bcad7701d75e4dd4 P 8b8b3ad3aa8f4053b2167dd550f2086f: No bootstrap required, opened a new log
I20240501 07:58:54.634241 7299 ts_tablet_manager.cc:1367] T 8480c7b51a914f03bcad7701d75e4dd4 P 8315d22c56f7439a9e1fc82f153f018c: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.001s
I20240501 07:58:54.634313 7301 ts_tablet_manager.cc:1367] T 8480c7b51a914f03bcad7701d75e4dd4 P 8b8b3ad3aa8f4053b2167dd550f2086f: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 07:58:54.634802 7298 raft_consensus.cc:348] T def178155f14416391e89ba972f4d723 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:58:54.634802 7301 raft_consensus.cc:348] T 8480c7b51a914f03bcad7701d75e4dd4 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } }
I20240501 07:58:54.634846 7299 raft_consensus.cc:348] T 8480c7b51a914f03bcad7701d75e4dd4 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } }
I20240501 07:58:54.634917 7301 raft_consensus.cc:374] T 8480c7b51a914f03bcad7701d75e4dd4 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:54.634914 7298 raft_consensus.cc:374] T def178155f14416391e89ba972f4d723 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:54.634939 7299 raft_consensus.cc:374] T 8480c7b51a914f03bcad7701d75e4dd4 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:54.634951 7301 raft_consensus.cc:729] T 8480c7b51a914f03bcad7701d75e4dd4 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b8b3ad3aa8f4053b2167dd550f2086f, State: Initialized, Role: FOLLOWER
I20240501 07:58:54.634951 7298 raft_consensus.cc:729] T def178155f14416391e89ba972f4d723 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b164cbbc00f4e8eb3254ddd349a3ff9, State: Initialized, Role: FOLLOWER
I20240501 07:58:54.634966 7299 raft_consensus.cc:729] T 8480c7b51a914f03bcad7701d75e4dd4 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8315d22c56f7439a9e1fc82f153f018c, State: Initialized, Role: FOLLOWER
I20240501 07:58:54.635071 7301 consensus_queue.cc:260] T 8480c7b51a914f03bcad7701d75e4dd4 P 8b8b3ad3aa8f4053b2167dd550f2086f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } }
I20240501 07:58:54.635071 7298 consensus_queue.cc:260] T def178155f14416391e89ba972f4d723 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:58:54.635107 7299 consensus_queue.cc:260] T 8480c7b51a914f03bcad7701d75e4dd4 P 8315d22c56f7439a9e1fc82f153f018c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } }
I20240501 07:58:54.635344 7298 ts_tablet_manager.cc:1398] T def178155f14416391e89ba972f4d723 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:58:54.635344 7301 ts_tablet_manager.cc:1398] T 8480c7b51a914f03bcad7701d75e4dd4 P 8b8b3ad3aa8f4053b2167dd550f2086f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:54.635375 7299 ts_tablet_manager.cc:1398] T 8480c7b51a914f03bcad7701d75e4dd4 P 8315d22c56f7439a9e1fc82f153f018c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:54.635514 7299 tablet_bootstrap.cc:492] T def178155f14416391e89ba972f4d723 P 8315d22c56f7439a9e1fc82f153f018c: Bootstrap starting.
I20240501 07:58:54.636694 7299 tablet_bootstrap.cc:654] T def178155f14416391e89ba972f4d723 P 8315d22c56f7439a9e1fc82f153f018c: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:54.637883 7304 tablet_bootstrap.cc:492] T def178155f14416391e89ba972f4d723 P 1f9cb72be98941b5a8e504cbe34132b9: No bootstrap required, opened a new log
I20240501 07:58:54.637920 7299 tablet_bootstrap.cc:492] T def178155f14416391e89ba972f4d723 P 8315d22c56f7439a9e1fc82f153f018c: No bootstrap required, opened a new log
I20240501 07:58:54.637990 7299 ts_tablet_manager.cc:1367] T def178155f14416391e89ba972f4d723 P 8315d22c56f7439a9e1fc82f153f018c: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240501 07:58:54.638149 7304 ts_tablet_manager.cc:1367] T def178155f14416391e89ba972f4d723 P 1f9cb72be98941b5a8e504cbe34132b9: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20240501 07:58:54.638469 7299 raft_consensus.cc:348] T def178155f14416391e89ba972f4d723 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:58:54.638588 7299 raft_consensus.cc:374] T def178155f14416391e89ba972f4d723 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:54.638620 7299 raft_consensus.cc:729] T def178155f14416391e89ba972f4d723 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8315d22c56f7439a9e1fc82f153f018c, State: Initialized, Role: FOLLOWER
I20240501 07:58:54.638741 7299 consensus_queue.cc:260] T def178155f14416391e89ba972f4d723 P 8315d22c56f7439a9e1fc82f153f018c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:58:54.639055 7299 ts_tablet_manager.cc:1398] T def178155f14416391e89ba972f4d723 P 8315d22c56f7439a9e1fc82f153f018c: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 07:58:54.641254 7304 raft_consensus.cc:348] T def178155f14416391e89ba972f4d723 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:58:54.641422 7304 raft_consensus.cc:374] T def178155f14416391e89ba972f4d723 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:54.641475 7304 raft_consensus.cc:729] T def178155f14416391e89ba972f4d723 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f9cb72be98941b5a8e504cbe34132b9, State: Initialized, Role: FOLLOWER
I20240501 07:58:54.641880 7304 consensus_queue.cc:260] T def178155f14416391e89ba972f4d723 P 1f9cb72be98941b5a8e504cbe34132b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:58:54.642572 7304 ts_tablet_manager.cc:1398] T def178155f14416391e89ba972f4d723 P 1f9cb72be98941b5a8e504cbe34132b9: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20240501 07:58:54.642781 7304 tablet_bootstrap.cc:492] T e04fe36b8efa4f9584e093279f7addf4 P 1f9cb72be98941b5a8e504cbe34132b9: Bootstrap starting.
I20240501 07:58:54.643509 7304 tablet_bootstrap.cc:654] T e04fe36b8efa4f9584e093279f7addf4 P 1f9cb72be98941b5a8e504cbe34132b9: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:54.644483 7304 tablet_bootstrap.cc:492] T e04fe36b8efa4f9584e093279f7addf4 P 1f9cb72be98941b5a8e504cbe34132b9: No bootstrap required, opened a new log
I20240501 07:58:54.644562 7304 ts_tablet_manager.cc:1367] T e04fe36b8efa4f9584e093279f7addf4 P 1f9cb72be98941b5a8e504cbe34132b9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:58:54.645023 7304 raft_consensus.cc:348] T e04fe36b8efa4f9584e093279f7addf4 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } }
I20240501 07:58:54.645105 7304 raft_consensus.cc:374] T e04fe36b8efa4f9584e093279f7addf4 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:54.645139 7304 raft_consensus.cc:729] T e04fe36b8efa4f9584e093279f7addf4 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f9cb72be98941b5a8e504cbe34132b9, State: Initialized, Role: FOLLOWER
I20240501 07:58:54.645263 7304 consensus_queue.cc:260] T e04fe36b8efa4f9584e093279f7addf4 P 1f9cb72be98941b5a8e504cbe34132b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } }
I20240501 07:58:54.645520 7304 ts_tablet_manager.cc:1398] T e04fe36b8efa4f9584e093279f7addf4 P 1f9cb72be98941b5a8e504cbe34132b9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:54.645581 7306 raft_consensus.cc:482] T e04fe36b8efa4f9584e093279f7addf4 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:54.645753 7306 raft_consensus.cc:504] T e04fe36b8efa4f9584e093279f7addf4 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } }
I20240501 07:58:54.646917 7306 leader_election.cc:288] T e04fe36b8efa4f9584e093279f7addf4 P 8b8b3ad3aa8f4053b2167dd550f2086f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132:36773), 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130:32941)
I20240501 07:58:54.650743 6708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e04fe36b8efa4f9584e093279f7addf4" candidate_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f9cb72be98941b5a8e504cbe34132b9" is_pre_election: true
I20240501 07:58:54.650987 6708 raft_consensus.cc:2454] T e04fe36b8efa4f9584e093279f7addf4 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b8b3ad3aa8f4053b2167dd550f2086f in term 0.
I20240501 07:58:54.650976 6446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e04fe36b8efa4f9584e093279f7addf4" candidate_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8315d22c56f7439a9e1fc82f153f018c" is_pre_election: true
I20240501 07:58:54.651216 6446 raft_consensus.cc:2454] T e04fe36b8efa4f9584e093279f7addf4 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b8b3ad3aa8f4053b2167dd550f2086f in term 0.
I20240501 07:58:54.651360 6250 leader_election.cc:302] T e04fe36b8efa4f9584e093279f7addf4 P 8b8b3ad3aa8f4053b2167dd550f2086f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f9cb72be98941b5a8e504cbe34132b9, 8b8b3ad3aa8f4053b2167dd550f2086f; no voters:
I20240501 07:58:54.651614 7306 raft_consensus.cc:2781] T e04fe36b8efa4f9584e093279f7addf4 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:54.651679 7306 raft_consensus.cc:482] T e04fe36b8efa4f9584e093279f7addf4 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:54.651708 7306 raft_consensus.cc:3037] T e04fe36b8efa4f9584e093279f7addf4 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:54.652558 7306 raft_consensus.cc:504] T e04fe36b8efa4f9584e093279f7addf4 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } }
I20240501 07:58:54.652876 7306 leader_election.cc:288] T e04fe36b8efa4f9584e093279f7addf4 P 8b8b3ad3aa8f4053b2167dd550f2086f [CANDIDATE]: Term 1 election: Requested vote from peers 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132:36773), 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130:32941)
I20240501 07:58:54.653231 6708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e04fe36b8efa4f9584e093279f7addf4" candidate_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f9cb72be98941b5a8e504cbe34132b9"
I20240501 07:58:54.653314 6446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e04fe36b8efa4f9584e093279f7addf4" candidate_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8315d22c56f7439a9e1fc82f153f018c"
I20240501 07:58:54.653367 6708 raft_consensus.cc:3037] T e04fe36b8efa4f9584e093279f7addf4 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:54.653400 6446 raft_consensus.cc:3037] T e04fe36b8efa4f9584e093279f7addf4 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:54.654181 6708 raft_consensus.cc:2454] T e04fe36b8efa4f9584e093279f7addf4 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b8b3ad3aa8f4053b2167dd550f2086f in term 1.
I20240501 07:58:54.654453 6250 leader_election.cc:302] T e04fe36b8efa4f9584e093279f7addf4 P 8b8b3ad3aa8f4053b2167dd550f2086f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f9cb72be98941b5a8e504cbe34132b9, 8b8b3ad3aa8f4053b2167dd550f2086f; no voters:
I20240501 07:58:54.654544 6446 raft_consensus.cc:2454] T e04fe36b8efa4f9584e093279f7addf4 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b8b3ad3aa8f4053b2167dd550f2086f in term 1.
I20240501 07:58:54.654647 7306 raft_consensus.cc:2781] T e04fe36b8efa4f9584e093279f7addf4 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:54.654834 7306 raft_consensus.cc:686] T e04fe36b8efa4f9584e093279f7addf4 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 LEADER]: Becoming Leader. State: Replica: 8b8b3ad3aa8f4053b2167dd550f2086f, State: Running, Role: LEADER
I20240501 07:58:54.655124 7306 consensus_queue.cc:237] T e04fe36b8efa4f9584e093279f7addf4 P 8b8b3ad3aa8f4053b2167dd550f2086f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } }
I20240501 07:58:54.658105 6172 catalog_manager.cc:5484] T e04fe36b8efa4f9584e093279f7addf4 P 8b8b3ad3aa8f4053b2167dd550f2086f reported cstate change: term changed from 0 to 1, leader changed from <none> to 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129). New cstate: current_term: 1 leader_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } health_report { overall_health: UNKNOWN } } }
W20240501 07:58:54.693856 6362 tablet.cc:2369] T 8480c7b51a914f03bcad7701d75e4dd4 P 8b8b3ad3aa8f4053b2167dd550f2086f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 07:58:54.696703 7305 raft_consensus.cc:482] T def178155f14416391e89ba972f4d723 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:54.696906 7305 raft_consensus.cc:504] T def178155f14416391e89ba972f4d723 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:58:54.698093 7305 leader_election.cc:288] T def178155f14416391e89ba972f4d723 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130:32941), 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132:36773)
I20240501 07:58:54.701895 6708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "def178155f14416391e89ba972f4d723" candidate_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f9cb72be98941b5a8e504cbe34132b9" is_pre_election: true
I20240501 07:58:54.701895 6446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "def178155f14416391e89ba972f4d723" candidate_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8315d22c56f7439a9e1fc82f153f018c" is_pre_election: true
I20240501 07:58:54.702029 6446 raft_consensus.cc:2454] T def178155f14416391e89ba972f4d723 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b164cbbc00f4e8eb3254ddd349a3ff9 in term 0.
I20240501 07:58:54.702047 6708 raft_consensus.cc:2454] T def178155f14416391e89ba972f4d723 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b164cbbc00f4e8eb3254ddd349a3ff9 in term 0.
I20240501 07:58:54.702389 6512 leader_election.cc:302] T def178155f14416391e89ba972f4d723 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f9cb72be98941b5a8e504cbe34132b9, 3b164cbbc00f4e8eb3254ddd349a3ff9; no voters:
I20240501 07:58:54.702652 7305 raft_consensus.cc:2781] T def178155f14416391e89ba972f4d723 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:54.702720 7305 raft_consensus.cc:482] T def178155f14416391e89ba972f4d723 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:54.702749 7305 raft_consensus.cc:3037] T def178155f14416391e89ba972f4d723 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:54.703625 7305 raft_consensus.cc:504] T def178155f14416391e89ba972f4d723 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:58:54.703928 7305 leader_election.cc:288] T def178155f14416391e89ba972f4d723 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [CANDIDATE]: Term 1 election: Requested vote from peers 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130:32941), 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132:36773)
I20240501 07:58:54.704277 6446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "def178155f14416391e89ba972f4d723" candidate_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8315d22c56f7439a9e1fc82f153f018c"
I20240501 07:58:54.704334 6708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "def178155f14416391e89ba972f4d723" candidate_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f9cb72be98941b5a8e504cbe34132b9"
I20240501 07:58:54.704401 6446 raft_consensus.cc:3037] T def178155f14416391e89ba972f4d723 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:54.704432 6708 raft_consensus.cc:3037] T def178155f14416391e89ba972f4d723 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:54.705372 6708 raft_consensus.cc:2454] T def178155f14416391e89ba972f4d723 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b164cbbc00f4e8eb3254ddd349a3ff9 in term 1.
I20240501 07:58:54.705372 6446 raft_consensus.cc:2454] T def178155f14416391e89ba972f4d723 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b164cbbc00f4e8eb3254ddd349a3ff9 in term 1.
I20240501 07:58:54.705679 6512 leader_election.cc:302] T def178155f14416391e89ba972f4d723 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f9cb72be98941b5a8e504cbe34132b9, 3b164cbbc00f4e8eb3254ddd349a3ff9; no voters:
I20240501 07:58:54.705896 7305 raft_consensus.cc:2781] T def178155f14416391e89ba972f4d723 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:54.706104 7305 raft_consensus.cc:686] T def178155f14416391e89ba972f4d723 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 LEADER]: Becoming Leader. State: Replica: 3b164cbbc00f4e8eb3254ddd349a3ff9, State: Running, Role: LEADER
I20240501 07:58:54.706391 7305 consensus_queue.cc:237] T def178155f14416391e89ba972f4d723 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:58:54.709374 6172 catalog_manager.cc:5484] T def178155f14416391e89ba972f4d723 P 3b164cbbc00f4e8eb3254ddd349a3ff9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } health_report { overall_health: UNKNOWN } } }
W20240501 07:58:54.756247 6755 tablet.cc:2369] T def178155f14416391e89ba972f4d723 P 1f9cb72be98941b5a8e504cbe34132b9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 07:58:54.785512 7305 raft_consensus.cc:482] T 8480c7b51a914f03bcad7701d75e4dd4 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:54.785656 7305 raft_consensus.cc:504] T 8480c7b51a914f03bcad7701d75e4dd4 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } }
I20240501 07:58:54.786168 7305 leader_election.cc:288] T 8480c7b51a914f03bcad7701d75e4dd4 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129:46875), 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130:32941)
I20240501 07:58:54.786708 6446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8480c7b51a914f03bcad7701d75e4dd4" candidate_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8315d22c56f7439a9e1fc82f153f018c" is_pre_election: true
I20240501 07:58:54.786916 6446 raft_consensus.cc:2454] T 8480c7b51a914f03bcad7701d75e4dd4 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b164cbbc00f4e8eb3254ddd349a3ff9 in term 0.
I20240501 07:58:54.787242 6512 leader_election.cc:302] T 8480c7b51a914f03bcad7701d75e4dd4 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b164cbbc00f4e8eb3254ddd349a3ff9, 8315d22c56f7439a9e1fc82f153f018c; no voters:
I20240501 07:58:54.787428 7305 raft_consensus.cc:2781] T 8480c7b51a914f03bcad7701d75e4dd4 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:54.787492 7305 raft_consensus.cc:482] T 8480c7b51a914f03bcad7701d75e4dd4 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:54.787544 7305 raft_consensus.cc:3037] T 8480c7b51a914f03bcad7701d75e4dd4 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:54.788436 7305 raft_consensus.cc:504] T 8480c7b51a914f03bcad7701d75e4dd4 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } }
I20240501 07:58:54.788797 7305 leader_election.cc:288] T 8480c7b51a914f03bcad7701d75e4dd4 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [CANDIDATE]: Term 1 election: Requested vote from peers 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129:46875), 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130:32941)
I20240501 07:58:54.789188 6446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8480c7b51a914f03bcad7701d75e4dd4" candidate_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8315d22c56f7439a9e1fc82f153f018c"
I20240501 07:58:54.789315 6446 raft_consensus.cc:3037] T 8480c7b51a914f03bcad7701d75e4dd4 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:54.790153 6446 raft_consensus.cc:2454] T 8480c7b51a914f03bcad7701d75e4dd4 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b164cbbc00f4e8eb3254ddd349a3ff9 in term 1.
I20240501 07:58:54.790159 6315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8480c7b51a914f03bcad7701d75e4dd4" candidate_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" is_pre_election: true
I20240501 07:58:54.790351 6314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8480c7b51a914f03bcad7701d75e4dd4" candidate_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f"
I20240501 07:58:54.790364 6315 raft_consensus.cc:2454] T 8480c7b51a914f03bcad7701d75e4dd4 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b164cbbc00f4e8eb3254ddd349a3ff9 in term 0.
I20240501 07:58:54.790450 6512 leader_election.cc:302] T 8480c7b51a914f03bcad7701d75e4dd4 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b164cbbc00f4e8eb3254ddd349a3ff9, 8315d22c56f7439a9e1fc82f153f018c; no voters:
I20240501 07:58:54.790781 7305 raft_consensus.cc:2781] T 8480c7b51a914f03bcad7701d75e4dd4 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:54.790879 7305 raft_consensus.cc:686] T 8480c7b51a914f03bcad7701d75e4dd4 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 LEADER]: Becoming Leader. State: Replica: 3b164cbbc00f4e8eb3254ddd349a3ff9, State: Running, Role: LEADER
I20240501 07:58:54.791020 7305 consensus_queue.cc:237] T 8480c7b51a914f03bcad7701d75e4dd4 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } }
I20240501 07:58:54.792584 6172 catalog_manager.cc:5484] T 8480c7b51a914f03bcad7701d75e4dd4 P 3b164cbbc00f4e8eb3254ddd349a3ff9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } health_report { overall_health: HEALTHY } } }
W20240501 07:58:54.823695 6624 tablet.cc:2369] T 8480c7b51a914f03bcad7701d75e4dd4 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 07:58:54.876756 6493 tablet.cc:2369] T 8480c7b51a914f03bcad7701d75e4dd4 P 8315d22c56f7439a9e1fc82f153f018c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 07:58:54.903121 7324 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:58:54.903414 7324 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:58:54.906584 7324 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
I20240501 07:58:55.000317 6708 raft_consensus.cc:1261] T def178155f14416391e89ba972f4d723 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Refusing update from remote peer 3b164cbbc00f4e8eb3254ddd349a3ff9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:55.001396 6446 raft_consensus.cc:1261] T def178155f14416391e89ba972f4d723 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Refusing update from remote peer 3b164cbbc00f4e8eb3254ddd349a3ff9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:55.002070 7305 consensus_queue.cc:1035] T def178155f14416391e89ba972f4d723 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:55.002517 7305 consensus_queue.cc:1035] T def178155f14416391e89ba972f4d723 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:55.003764 6314 raft_consensus.cc:3037] T 8480c7b51a914f03bcad7701d75e4dd4 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:55.004093 6445 raft_consensus.cc:1261] T 8480c7b51a914f03bcad7701d75e4dd4 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Refusing update from remote peer 3b164cbbc00f4e8eb3254ddd349a3ff9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:55.004777 6445 raft_consensus.cc:1261] T e04fe36b8efa4f9584e093279f7addf4 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Refusing update from remote peer 8b8b3ad3aa8f4053b2167dd550f2086f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:55.005049 6314 raft_consensus.cc:1261] T 8480c7b51a914f03bcad7701d75e4dd4 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Refusing update from remote peer 3b164cbbc00f4e8eb3254ddd349a3ff9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:55.005249 7317 consensus_queue.cc:1035] T e04fe36b8efa4f9584e093279f7addf4 P 8b8b3ad3aa8f4053b2167dd550f2086f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:55.005668 6707 raft_consensus.cc:1261] T e04fe36b8efa4f9584e093279f7addf4 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Refusing update from remote peer 8b8b3ad3aa8f4053b2167dd550f2086f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:55.010793 7355 consensus_queue.cc:1035] T 8480c7b51a914f03bcad7701d75e4dd4 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:55.011917 7317 consensus_queue.cc:1035] T e04fe36b8efa4f9584e093279f7addf4 P 8b8b3ad3aa8f4053b2167dd550f2086f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:55.012584 7321 consensus_queue.cc:1035] T 8480c7b51a914f03bcad7701d75e4dd4 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:55.013860 7354 mvcc.cc:205] Tried to move back new op lower bound from 7022798172153724928 to 7022798170960027648. Current Snapshot: MvccSnapshot[applied={T|T < 7022798172153724928}]
I20240501 07:58:55.023161 7356 mvcc.cc:205] Tried to move back new op lower bound from 7022798172153724928 to 7022798170960027648. Current Snapshot: MvccSnapshot[applied={T|T < 7022798172153724928}]
I20240501 07:58:55.035048 7352 mvcc.cc:205] Tried to move back new op lower bound from 7022798172187955200 to 7022798170749497344. Current Snapshot: MvccSnapshot[applied={T|T < 7022798172166987776}]
INSERT report
rows total: 2000
time total: 115.087 ms
time per row: 0.0575433 ms
I20240501 07:58:55.060176 7358 mvcc.cc:205] Tried to move back new op lower bound from 7022798172187955200 to 7022798170749497344. Current Snapshot: MvccSnapshot[applied={T|T < 7022798172166987776}]
I20240501 07:58:55.071838 6169 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55950:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 07:58:55.076948 6557 tablet_service.cc:1467] Processing CreateTablet for tablet ff9dc384dbac47dc9795617c64688943 (DEFAULT_TABLE table=rebalance_test_table_1 [id=d0e1f982440849beb494081bea9cb211]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:55.077173 6687 tablet_service.cc:1467] Processing CreateTablet for tablet ff9dc384dbac47dc9795617c64688943 (DEFAULT_TABLE table=rebalance_test_table_1 [id=d0e1f982440849beb494081bea9cb211]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:55.077201 6295 tablet_service.cc:1467] Processing CreateTablet for tablet ff9dc384dbac47dc9795617c64688943 (DEFAULT_TABLE table=rebalance_test_table_1 [id=d0e1f982440849beb494081bea9cb211]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:55.077338 6557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff9dc384dbac47dc9795617c64688943. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:55.077414 6687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff9dc384dbac47dc9795617c64688943. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:55.077503 6295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff9dc384dbac47dc9795617c64688943. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:55.079353 7301 tablet_bootstrap.cc:492] T ff9dc384dbac47dc9795617c64688943 P 8b8b3ad3aa8f4053b2167dd550f2086f: Bootstrap starting.
I20240501 07:58:55.079509 7304 tablet_bootstrap.cc:492] T ff9dc384dbac47dc9795617c64688943 P 1f9cb72be98941b5a8e504cbe34132b9: Bootstrap starting.
I20240501 07:58:55.079509 7298 tablet_bootstrap.cc:492] T ff9dc384dbac47dc9795617c64688943 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Bootstrap starting.
I20240501 07:58:55.080304 7304 tablet_bootstrap.cc:654] T ff9dc384dbac47dc9795617c64688943 P 1f9cb72be98941b5a8e504cbe34132b9: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:55.080304 7298 tablet_bootstrap.cc:654] T ff9dc384dbac47dc9795617c64688943 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:55.080325 7301 tablet_bootstrap.cc:654] T ff9dc384dbac47dc9795617c64688943 P 8b8b3ad3aa8f4053b2167dd550f2086f: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:55.080472 6294 tablet_service.cc:1467] Processing CreateTablet for tablet 3745acee2b5543b697a3c8f45d2618bb (DEFAULT_TABLE table=rebalance_test_table_1 [id=d0e1f982440849beb494081bea9cb211]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:55.080698 6294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3745acee2b5543b697a3c8f45d2618bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:55.081240 6426 tablet_service.cc:1467] Processing CreateTablet for tablet 3745acee2b5543b697a3c8f45d2618bb (DEFAULT_TABLE table=rebalance_test_table_1 [id=d0e1f982440849beb494081bea9cb211]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:55.081488 6426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3745acee2b5543b697a3c8f45d2618bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:55.081496 6556 tablet_service.cc:1467] Processing CreateTablet for tablet 3745acee2b5543b697a3c8f45d2618bb (DEFAULT_TABLE table=rebalance_test_table_1 [id=d0e1f982440849beb494081bea9cb211]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:55.081717 6556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3745acee2b5543b697a3c8f45d2618bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:55.081845 6557 tablet_service.cc:1467] Processing CreateTablet for tablet 6e91683ff22243c1b3877733d74e2886 (DEFAULT_TABLE table=rebalance_test_table_1 [id=d0e1f982440849beb494081bea9cb211]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:55.082056 6557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e91683ff22243c1b3877733d74e2886. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:55.083703 7299 tablet_bootstrap.cc:492] T 3745acee2b5543b697a3c8f45d2618bb P 8315d22c56f7439a9e1fc82f153f018c: Bootstrap starting.
I20240501 07:58:55.084633 7299 tablet_bootstrap.cc:654] T 3745acee2b5543b697a3c8f45d2618bb P 8315d22c56f7439a9e1fc82f153f018c: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:55.084668 7298 tablet_bootstrap.cc:492] T ff9dc384dbac47dc9795617c64688943 P 3b164cbbc00f4e8eb3254ddd349a3ff9: No bootstrap required, opened a new log
I20240501 07:58:55.084743 7298 ts_tablet_manager.cc:1367] T ff9dc384dbac47dc9795617c64688943 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240501 07:58:55.084705 6295 tablet_service.cc:1467] Processing CreateTablet for tablet 6e91683ff22243c1b3877733d74e2886 (DEFAULT_TABLE table=rebalance_test_table_1 [id=d0e1f982440849beb494081bea9cb211]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:55.084930 6295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e91683ff22243c1b3877733d74e2886. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:55.085083 7301 tablet_bootstrap.cc:492] T ff9dc384dbac47dc9795617c64688943 P 8b8b3ad3aa8f4053b2167dd550f2086f: No bootstrap required, opened a new log
I20240501 07:58:55.085158 7301 ts_tablet_manager.cc:1367] T ff9dc384dbac47dc9795617c64688943 P 8b8b3ad3aa8f4053b2167dd550f2086f: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 07:58:55.085222 7298 raft_consensus.cc:348] T ff9dc384dbac47dc9795617c64688943 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:58:55.085318 7298 raft_consensus.cc:374] T ff9dc384dbac47dc9795617c64688943 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:55.085346 7298 raft_consensus.cc:729] T ff9dc384dbac47dc9795617c64688943 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b164cbbc00f4e8eb3254ddd349a3ff9, State: Initialized, Role: FOLLOWER
I20240501 07:58:55.085458 7298 consensus_queue.cc:260] T ff9dc384dbac47dc9795617c64688943 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:58:55.085531 7304 tablet_bootstrap.cc:492] T ff9dc384dbac47dc9795617c64688943 P 1f9cb72be98941b5a8e504cbe34132b9: No bootstrap required, opened a new log
I20240501 07:58:55.085611 7304 ts_tablet_manager.cc:1367] T ff9dc384dbac47dc9795617c64688943 P 1f9cb72be98941b5a8e504cbe34132b9: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 07:58:55.085601 7301 raft_consensus.cc:348] T ff9dc384dbac47dc9795617c64688943 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:58:55.085697 7301 raft_consensus.cc:374] T ff9dc384dbac47dc9795617c64688943 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:55.085717 7298 ts_tablet_manager.cc:1398] T ff9dc384dbac47dc9795617c64688943 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:55.085732 7301 raft_consensus.cc:729] T ff9dc384dbac47dc9795617c64688943 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b8b3ad3aa8f4053b2167dd550f2086f, State: Initialized, Role: FOLLOWER
I20240501 07:58:55.085845 7301 consensus_queue.cc:260] T ff9dc384dbac47dc9795617c64688943 P 8b8b3ad3aa8f4053b2167dd550f2086f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:58:55.085903 7298 tablet_bootstrap.cc:492] T 3745acee2b5543b697a3c8f45d2618bb P 3b164cbbc00f4e8eb3254ddd349a3ff9: Bootstrap starting.
I20240501 07:58:55.086107 7301 ts_tablet_manager.cc:1398] T ff9dc384dbac47dc9795617c64688943 P 8b8b3ad3aa8f4053b2167dd550f2086f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:55.086045 7304 raft_consensus.cc:348] T ff9dc384dbac47dc9795617c64688943 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:58:55.086146 7304 raft_consensus.cc:374] T ff9dc384dbac47dc9795617c64688943 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:55.086177 7304 raft_consensus.cc:729] T ff9dc384dbac47dc9795617c64688943 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f9cb72be98941b5a8e504cbe34132b9, State: Initialized, Role: FOLLOWER
I20240501 07:58:55.086279 7301 tablet_bootstrap.cc:492] T 3745acee2b5543b697a3c8f45d2618bb P 8b8b3ad3aa8f4053b2167dd550f2086f: Bootstrap starting.
I20240501 07:58:55.086306 7304 consensus_queue.cc:260] T ff9dc384dbac47dc9795617c64688943 P 1f9cb72be98941b5a8e504cbe34132b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:58:55.086546 7304 ts_tablet_manager.cc:1398] T ff9dc384dbac47dc9795617c64688943 P 1f9cb72be98941b5a8e504cbe34132b9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:55.086743 7298 tablet_bootstrap.cc:654] T 3745acee2b5543b697a3c8f45d2618bb P 3b164cbbc00f4e8eb3254ddd349a3ff9: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:55.087489 7301 tablet_bootstrap.cc:654] T 3745acee2b5543b697a3c8f45d2618bb P 8b8b3ad3aa8f4053b2167dd550f2086f: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:55.087474 6687 tablet_service.cc:1467] Processing CreateTablet for tablet 6e91683ff22243c1b3877733d74e2886 (DEFAULT_TABLE table=rebalance_test_table_1 [id=d0e1f982440849beb494081bea9cb211]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:55.087641 7299 tablet_bootstrap.cc:492] T 3745acee2b5543b697a3c8f45d2618bb P 8315d22c56f7439a9e1fc82f153f018c: No bootstrap required, opened a new log
I20240501 07:58:55.087698 6687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e91683ff22243c1b3877733d74e2886. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:55.087718 7299 ts_tablet_manager.cc:1367] T 3745acee2b5543b697a3c8f45d2618bb P 8315d22c56f7439a9e1fc82f153f018c: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240501 07:58:55.088166 7299 raft_consensus.cc:348] T 3745acee2b5543b697a3c8f45d2618bb P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } }
I20240501 07:58:55.088244 7299 raft_consensus.cc:374] T 3745acee2b5543b697a3c8f45d2618bb P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:55.088265 7299 raft_consensus.cc:729] T 3745acee2b5543b697a3c8f45d2618bb P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8315d22c56f7439a9e1fc82f153f018c, State: Initialized, Role: FOLLOWER
I20240501 07:58:55.088366 7299 consensus_queue.cc:260] T 3745acee2b5543b697a3c8f45d2618bb P 8315d22c56f7439a9e1fc82f153f018c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } }
I20240501 07:58:55.088598 7299 ts_tablet_manager.cc:1398] T 3745acee2b5543b697a3c8f45d2618bb P 8315d22c56f7439a9e1fc82f153f018c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:58:55.090912 7304 tablet_bootstrap.cc:492] T 6e91683ff22243c1b3877733d74e2886 P 1f9cb72be98941b5a8e504cbe34132b9: Bootstrap starting.
I20240501 07:58:55.091675 7298 tablet_bootstrap.cc:492] T 3745acee2b5543b697a3c8f45d2618bb P 3b164cbbc00f4e8eb3254ddd349a3ff9: No bootstrap required, opened a new log
I20240501 07:58:55.091724 7304 tablet_bootstrap.cc:654] T 6e91683ff22243c1b3877733d74e2886 P 1f9cb72be98941b5a8e504cbe34132b9: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:55.091773 7298 ts_tablet_manager.cc:1367] T 3745acee2b5543b697a3c8f45d2618bb P 3b164cbbc00f4e8eb3254ddd349a3ff9: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 07:58:55.091816 7301 tablet_bootstrap.cc:492] T 3745acee2b5543b697a3c8f45d2618bb P 8b8b3ad3aa8f4053b2167dd550f2086f: No bootstrap required, opened a new log
I20240501 07:58:55.091903 7301 ts_tablet_manager.cc:1367] T 3745acee2b5543b697a3c8f45d2618bb P 8b8b3ad3aa8f4053b2167dd550f2086f: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 07:58:55.092222 7298 raft_consensus.cc:348] T 3745acee2b5543b697a3c8f45d2618bb P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } }
I20240501 07:58:55.092267 7301 raft_consensus.cc:348] T 3745acee2b5543b697a3c8f45d2618bb P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } }
I20240501 07:58:55.092334 7298 raft_consensus.cc:374] T 3745acee2b5543b697a3c8f45d2618bb P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:55.092370 7298 raft_consensus.cc:729] T 3745acee2b5543b697a3c8f45d2618bb P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b164cbbc00f4e8eb3254ddd349a3ff9, State: Initialized, Role: FOLLOWER
I20240501 07:58:55.092367 7301 raft_consensus.cc:374] T 3745acee2b5543b697a3c8f45d2618bb P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:55.092409 7301 raft_consensus.cc:729] T 3745acee2b5543b697a3c8f45d2618bb P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b8b3ad3aa8f4053b2167dd550f2086f, State: Initialized, Role: FOLLOWER
I20240501 07:58:55.092552 7301 consensus_queue.cc:260] T 3745acee2b5543b697a3c8f45d2618bb P 8b8b3ad3aa8f4053b2167dd550f2086f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } }
I20240501 07:58:55.092621 7298 consensus_queue.cc:260] T 3745acee2b5543b697a3c8f45d2618bb P 3b164cbbc00f4e8eb3254ddd349a3ff9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } }
I20240501 07:58:55.092818 7304 tablet_bootstrap.cc:492] T 6e91683ff22243c1b3877733d74e2886 P 1f9cb72be98941b5a8e504cbe34132b9: No bootstrap required, opened a new log
I20240501 07:58:55.092892 7304 ts_tablet_manager.cc:1367] T 6e91683ff22243c1b3877733d74e2886 P 1f9cb72be98941b5a8e504cbe34132b9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:58:55.092900 7298 ts_tablet_manager.cc:1398] T 3745acee2b5543b697a3c8f45d2618bb P 3b164cbbc00f4e8eb3254ddd349a3ff9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:55.092904 7301 ts_tablet_manager.cc:1398] T 3745acee2b5543b697a3c8f45d2618bb P 8b8b3ad3aa8f4053b2167dd550f2086f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:55.093062 7298 tablet_bootstrap.cc:492] T 6e91683ff22243c1b3877733d74e2886 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Bootstrap starting.
I20240501 07:58:55.093070 7301 tablet_bootstrap.cc:492] T 6e91683ff22243c1b3877733d74e2886 P 8b8b3ad3aa8f4053b2167dd550f2086f: Bootstrap starting.
I20240501 07:58:55.093931 7298 tablet_bootstrap.cc:654] T 6e91683ff22243c1b3877733d74e2886 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:55.093987 7301 tablet_bootstrap.cc:654] T 6e91683ff22243c1b3877733d74e2886 P 8b8b3ad3aa8f4053b2167dd550f2086f: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:55.094388 7304 raft_consensus.cc:348] T 6e91683ff22243c1b3877733d74e2886 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:58:55.094508 7304 raft_consensus.cc:374] T 6e91683ff22243c1b3877733d74e2886 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:55.094550 7304 raft_consensus.cc:729] T 6e91683ff22243c1b3877733d74e2886 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f9cb72be98941b5a8e504cbe34132b9, State: Initialized, Role: FOLLOWER
I20240501 07:58:55.094678 7304 consensus_queue.cc:260] T 6e91683ff22243c1b3877733d74e2886 P 1f9cb72be98941b5a8e504cbe34132b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:58:55.095069 7304 ts_tablet_manager.cc:1398] T 6e91683ff22243c1b3877733d74e2886 P 1f9cb72be98941b5a8e504cbe34132b9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 07:58:55.095536 7301 tablet_bootstrap.cc:492] T 6e91683ff22243c1b3877733d74e2886 P 8b8b3ad3aa8f4053b2167dd550f2086f: No bootstrap required, opened a new log
I20240501 07:58:55.095605 7298 tablet_bootstrap.cc:492] T 6e91683ff22243c1b3877733d74e2886 P 3b164cbbc00f4e8eb3254ddd349a3ff9: No bootstrap required, opened a new log
I20240501 07:58:55.095621 7301 ts_tablet_manager.cc:1367] T 6e91683ff22243c1b3877733d74e2886 P 8b8b3ad3aa8f4053b2167dd550f2086f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 07:58:55.095669 7298 ts_tablet_manager.cc:1367] T 6e91683ff22243c1b3877733d74e2886 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240501 07:58:55.096048 7298 raft_consensus.cc:348] T 6e91683ff22243c1b3877733d74e2886 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:58:55.096074 7301 raft_consensus.cc:348] T 6e91683ff22243c1b3877733d74e2886 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:58:55.096151 7298 raft_consensus.cc:374] T 6e91683ff22243c1b3877733d74e2886 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:55.096172 7301 raft_consensus.cc:374] T 6e91683ff22243c1b3877733d74e2886 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:55.096186 7298 raft_consensus.cc:729] T 6e91683ff22243c1b3877733d74e2886 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b164cbbc00f4e8eb3254ddd349a3ff9, State: Initialized, Role: FOLLOWER
I20240501 07:58:55.096205 7301 raft_consensus.cc:729] T 6e91683ff22243c1b3877733d74e2886 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b8b3ad3aa8f4053b2167dd550f2086f, State: Initialized, Role: FOLLOWER
I20240501 07:58:55.096316 7298 consensus_queue.cc:260] T 6e91683ff22243c1b3877733d74e2886 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:58:55.096343 7301 consensus_queue.cc:260] T 6e91683ff22243c1b3877733d74e2886 P 8b8b3ad3aa8f4053b2167dd550f2086f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:58:55.096598 7298 ts_tablet_manager.cc:1398] T 6e91683ff22243c1b3877733d74e2886 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:55.096640 7301 ts_tablet_manager.cc:1398] T 6e91683ff22243c1b3877733d74e2886 P 8b8b3ad3aa8f4053b2167dd550f2086f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:55.117110 7355 raft_consensus.cc:482] T 6e91683ff22243c1b3877733d74e2886 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:55.117242 7355 raft_consensus.cc:504] T 6e91683ff22243c1b3877733d74e2886 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:58:55.117671 7355 leader_election.cc:288] T 6e91683ff22243c1b3877733d74e2886 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129:46875), 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132:36773)
I20240501 07:58:55.118110 6708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e91683ff22243c1b3877733d74e2886" candidate_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f9cb72be98941b5a8e504cbe34132b9" is_pre_election: true
I20240501 07:58:55.118136 6314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e91683ff22243c1b3877733d74e2886" candidate_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" is_pre_election: true
I20240501 07:58:55.118274 6708 raft_consensus.cc:2454] T 6e91683ff22243c1b3877733d74e2886 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b164cbbc00f4e8eb3254ddd349a3ff9 in term 0.
I20240501 07:58:55.118288 6314 raft_consensus.cc:2454] T 6e91683ff22243c1b3877733d74e2886 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b164cbbc00f4e8eb3254ddd349a3ff9 in term 0.
I20240501 07:58:55.118597 6510 leader_election.cc:302] T 6e91683ff22243c1b3877733d74e2886 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b164cbbc00f4e8eb3254ddd349a3ff9, 8b8b3ad3aa8f4053b2167dd550f2086f; no voters:
I20240501 07:58:55.118770 7355 raft_consensus.cc:2781] T 6e91683ff22243c1b3877733d74e2886 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:55.118841 7355 raft_consensus.cc:482] T 6e91683ff22243c1b3877733d74e2886 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:55.118889 7355 raft_consensus.cc:3037] T 6e91683ff22243c1b3877733d74e2886 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:55.119688 7355 raft_consensus.cc:504] T 6e91683ff22243c1b3877733d74e2886 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:58:55.120011 7355 leader_election.cc:288] T 6e91683ff22243c1b3877733d74e2886 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [CANDIDATE]: Term 1 election: Requested vote from peers 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129:46875), 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132:36773)
I20240501 07:58:55.120340 6708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e91683ff22243c1b3877733d74e2886" candidate_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f9cb72be98941b5a8e504cbe34132b9"
I20240501 07:58:55.120333 6314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e91683ff22243c1b3877733d74e2886" candidate_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f"
I20240501 07:58:55.120441 6314 raft_consensus.cc:3037] T 6e91683ff22243c1b3877733d74e2886 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:55.120445 6708 raft_consensus.cc:3037] T 6e91683ff22243c1b3877733d74e2886 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:55.121258 6708 raft_consensus.cc:2454] T 6e91683ff22243c1b3877733d74e2886 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b164cbbc00f4e8eb3254ddd349a3ff9 in term 1.
I20240501 07:58:55.121371 6314 raft_consensus.cc:2454] T 6e91683ff22243c1b3877733d74e2886 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b164cbbc00f4e8eb3254ddd349a3ff9 in term 1.
I20240501 07:58:55.121541 6512 leader_election.cc:302] T 6e91683ff22243c1b3877733d74e2886 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f9cb72be98941b5a8e504cbe34132b9, 3b164cbbc00f4e8eb3254ddd349a3ff9; no voters:
I20240501 07:58:55.121707 7355 raft_consensus.cc:2781] T 6e91683ff22243c1b3877733d74e2886 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:55.121845 7355 raft_consensus.cc:686] T 6e91683ff22243c1b3877733d74e2886 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 LEADER]: Becoming Leader. State: Replica: 3b164cbbc00f4e8eb3254ddd349a3ff9, State: Running, Role: LEADER
I20240501 07:58:55.121985 7355 consensus_queue.cc:237] T 6e91683ff22243c1b3877733d74e2886 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:58:55.123473 6168 catalog_manager.cc:5484] T 6e91683ff22243c1b3877733d74e2886 P 3b164cbbc00f4e8eb3254ddd349a3ff9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } health_report { overall_health: UNKNOWN } } }
I20240501 07:58:55.197165 7355 raft_consensus.cc:482] T 3745acee2b5543b697a3c8f45d2618bb P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:55.197340 7355 raft_consensus.cc:504] T 3745acee2b5543b697a3c8f45d2618bb P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } }
I20240501 07:58:55.197818 7355 leader_election.cc:288] T 3745acee2b5543b697a3c8f45d2618bb P 3b164cbbc00f4e8eb3254ddd349a3ff9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130:32941), 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129:46875)
I20240501 07:58:55.198354 6314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3745acee2b5543b697a3c8f45d2618bb" candidate_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" is_pre_election: true
I20240501 07:58:55.198359 6446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3745acee2b5543b697a3c8f45d2618bb" candidate_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8315d22c56f7439a9e1fc82f153f018c" is_pre_election: true
I20240501 07:58:55.198521 6314 raft_consensus.cc:2454] T 3745acee2b5543b697a3c8f45d2618bb P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b164cbbc00f4e8eb3254ddd349a3ff9 in term 0.
I20240501 07:58:55.198536 6446 raft_consensus.cc:2454] T 3745acee2b5543b697a3c8f45d2618bb P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b164cbbc00f4e8eb3254ddd349a3ff9 in term 0.
I20240501 07:58:55.198865 6510 leader_election.cc:302] T 3745acee2b5543b697a3c8f45d2618bb P 3b164cbbc00f4e8eb3254ddd349a3ff9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b164cbbc00f4e8eb3254ddd349a3ff9, 8b8b3ad3aa8f4053b2167dd550f2086f; no voters:
I20240501 07:58:55.199047 7355 raft_consensus.cc:2781] T 3745acee2b5543b697a3c8f45d2618bb P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:55.199117 7355 raft_consensus.cc:482] T 3745acee2b5543b697a3c8f45d2618bb P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:55.199169 7355 raft_consensus.cc:3037] T 3745acee2b5543b697a3c8f45d2618bb P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:55.200007 7355 raft_consensus.cc:504] T 3745acee2b5543b697a3c8f45d2618bb P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } }
I20240501 07:58:55.200361 7355 leader_election.cc:288] T 3745acee2b5543b697a3c8f45d2618bb P 3b164cbbc00f4e8eb3254ddd349a3ff9 [CANDIDATE]: Term 1 election: Requested vote from peers 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130:32941), 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129:46875)
I20240501 07:58:55.200793 6446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3745acee2b5543b697a3c8f45d2618bb" candidate_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8315d22c56f7439a9e1fc82f153f018c"
I20240501 07:58:55.200811 6314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3745acee2b5543b697a3c8f45d2618bb" candidate_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f"
I20240501 07:58:55.200914 6314 raft_consensus.cc:3037] T 3745acee2b5543b697a3c8f45d2618bb P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:55.200918 6446 raft_consensus.cc:3037] T 3745acee2b5543b697a3c8f45d2618bb P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:55.201758 6314 raft_consensus.cc:2454] T 3745acee2b5543b697a3c8f45d2618bb P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b164cbbc00f4e8eb3254ddd349a3ff9 in term 1.
I20240501 07:58:55.201866 6446 raft_consensus.cc:2454] T 3745acee2b5543b697a3c8f45d2618bb P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b164cbbc00f4e8eb3254ddd349a3ff9 in term 1.
I20240501 07:58:55.202169 6512 leader_election.cc:302] T 3745acee2b5543b697a3c8f45d2618bb P 3b164cbbc00f4e8eb3254ddd349a3ff9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b164cbbc00f4e8eb3254ddd349a3ff9, 8315d22c56f7439a9e1fc82f153f018c; no voters:
I20240501 07:58:55.202355 7355 raft_consensus.cc:2781] T 3745acee2b5543b697a3c8f45d2618bb P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:55.202440 7355 raft_consensus.cc:686] T 3745acee2b5543b697a3c8f45d2618bb P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 LEADER]: Becoming Leader. State: Replica: 3b164cbbc00f4e8eb3254ddd349a3ff9, State: Running, Role: LEADER
I20240501 07:58:55.202564 7355 consensus_queue.cc:237] T 3745acee2b5543b697a3c8f45d2618bb P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } }
I20240501 07:58:55.204245 6168 catalog_manager.cc:5484] T 3745acee2b5543b697a3c8f45d2618bb P 3b164cbbc00f4e8eb3254ddd349a3ff9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } health_report { overall_health: UNKNOWN } } }
I20240501 07:58:55.345803 7313 raft_consensus.cc:482] T ff9dc384dbac47dc9795617c64688943 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:55.345954 7313 raft_consensus.cc:504] T ff9dc384dbac47dc9795617c64688943 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:58:55.347054 7313 leader_election.cc:288] T ff9dc384dbac47dc9795617c64688943 P 1f9cb72be98941b5a8e504cbe34132b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131:34471), 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129:46875)
I20240501 07:58:55.347442 7355 raft_consensus.cc:482] T ff9dc384dbac47dc9795617c64688943 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:55.347756 7355 raft_consensus.cc:504] T ff9dc384dbac47dc9795617c64688943 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:58:55.348609 6708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9dc384dbac47dc9795617c64688943" candidate_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f9cb72be98941b5a8e504cbe34132b9" is_pre_election: true
I20240501 07:58:55.348754 6708 raft_consensus.cc:2454] T ff9dc384dbac47dc9795617c64688943 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b164cbbc00f4e8eb3254ddd349a3ff9 in term 0.
I20240501 07:58:55.348704 6314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9dc384dbac47dc9795617c64688943" candidate_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" is_pre_election: true
I20240501 07:58:55.348857 6314 raft_consensus.cc:2454] T ff9dc384dbac47dc9795617c64688943 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b164cbbc00f4e8eb3254ddd349a3ff9 in term 0.
I20240501 07:58:55.349036 6512 leader_election.cc:302] T ff9dc384dbac47dc9795617c64688943 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f9cb72be98941b5a8e504cbe34132b9, 3b164cbbc00f4e8eb3254ddd349a3ff9; no voters:
I20240501 07:58:55.349186 7305 raft_consensus.cc:2781] T ff9dc384dbac47dc9795617c64688943 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:55.349236 7305 raft_consensus.cc:482] T ff9dc384dbac47dc9795617c64688943 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:55.349277 7305 raft_consensus.cc:3037] T ff9dc384dbac47dc9795617c64688943 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:55.349372 7355 leader_election.cc:288] T ff9dc384dbac47dc9795617c64688943 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129:46875), 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132:36773)
I20240501 07:58:55.350282 7305 raft_consensus.cc:504] T ff9dc384dbac47dc9795617c64688943 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:58:55.350569 7305 leader_election.cc:288] T ff9dc384dbac47dc9795617c64688943 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [CANDIDATE]: Term 1 election: Requested vote from peers 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129:46875), 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132:36773)
I20240501 07:58:55.350922 6708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9dc384dbac47dc9795617c64688943" candidate_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f9cb72be98941b5a8e504cbe34132b9"
I20240501 07:58:55.351027 6708 raft_consensus.cc:3037] T ff9dc384dbac47dc9795617c64688943 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:55.351807 6314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9dc384dbac47dc9795617c64688943" candidate_uuid: "1f9cb72be98941b5a8e504cbe34132b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" is_pre_election: true
I20240501 07:58:55.351919 6314 raft_consensus.cc:2454] T ff9dc384dbac47dc9795617c64688943 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f9cb72be98941b5a8e504cbe34132b9 in term 0.
I20240501 07:58:55.352075 6708 raft_consensus.cc:2454] T ff9dc384dbac47dc9795617c64688943 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b164cbbc00f4e8eb3254ddd349a3ff9 in term 1.
I20240501 07:58:55.352245 6641 leader_election.cc:302] T ff9dc384dbac47dc9795617c64688943 P 1f9cb72be98941b5a8e504cbe34132b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f9cb72be98941b5a8e504cbe34132b9, 8b8b3ad3aa8f4053b2167dd550f2086f; no voters:
I20240501 07:58:55.352335 6512 leader_election.cc:302] T ff9dc384dbac47dc9795617c64688943 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f9cb72be98941b5a8e504cbe34132b9, 3b164cbbc00f4e8eb3254ddd349a3ff9; no voters:
I20240501 07:58:55.352478 7305 raft_consensus.cc:2781] T ff9dc384dbac47dc9795617c64688943 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:55.352579 7305 raft_consensus.cc:686] T ff9dc384dbac47dc9795617c64688943 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 LEADER]: Becoming Leader. State: Replica: 3b164cbbc00f4e8eb3254ddd349a3ff9, State: Running, Role: LEADER
I20240501 07:58:55.352741 7305 consensus_queue.cc:237] T ff9dc384dbac47dc9795617c64688943 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:58:55.353238 6314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9dc384dbac47dc9795617c64688943" candidate_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f"
I20240501 07:58:55.353293 6577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9dc384dbac47dc9795617c64688943" candidate_uuid: "1f9cb72be98941b5a8e504cbe34132b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" is_pre_election: true
I20240501 07:58:55.353359 6314 raft_consensus.cc:3037] T ff9dc384dbac47dc9795617c64688943 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:55.353523 7313 raft_consensus.cc:2741] T ff9dc384dbac47dc9795617c64688943 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20240501 07:58:55.354331 6314 raft_consensus.cc:2454] T ff9dc384dbac47dc9795617c64688943 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b164cbbc00f4e8eb3254ddd349a3ff9 in term 1.
I20240501 07:58:55.354892 6168 catalog_manager.cc:5484] T ff9dc384dbac47dc9795617c64688943 P 3b164cbbc00f4e8eb3254ddd349a3ff9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } health_report { overall_health: UNKNOWN } } }
W20240501 07:58:55.495534 7399 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:58:55.495810 7399 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:58:55.499138 7399 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
I20240501 07:58:55.529796 7355 consensus_queue.cc:1035] T 6e91683ff22243c1b3877733d74e2886 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:55.534973 7321 consensus_queue.cc:1035] T 6e91683ff22243c1b3877733d74e2886 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:55.585139 7321 consensus_queue.cc:1035] T 3745acee2b5543b697a3c8f45d2618bb P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:55.590204 6708 raft_consensus.cc:1261] T ff9dc384dbac47dc9795617c64688943 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Refusing update from remote peer 3b164cbbc00f4e8eb3254ddd349a3ff9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:55.590827 7321 consensus_queue.cc:1035] T ff9dc384dbac47dc9795617c64688943 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:55.592062 7355 consensus_queue.cc:1035] T 3745acee2b5543b697a3c8f45d2618bb P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:55.595597 6314 raft_consensus.cc:1261] T ff9dc384dbac47dc9795617c64688943 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Refusing update from remote peer 3b164cbbc00f4e8eb3254ddd349a3ff9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:55.597066 7305 consensus_queue.cc:1035] T ff9dc384dbac47dc9795617c64688943 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 112.167 ms
time per row: 0.0560836 ms
I20240501 07:58:55.658871 6173 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55950:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 07:58:55.664762 6426 tablet_service.cc:1467] Processing CreateTablet for tablet 460b4d38c63a4a4ca177a37be70d927f (DEFAULT_TABLE table=rebalance_test_table_2 [id=a8027b31a8b34320800f512d75105639]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:55.665120 6426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 460b4d38c63a4a4ca177a37be70d927f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:55.665164 6424 tablet_service.cc:1467] Processing CreateTablet for tablet 7d63a75f2194484f8a59709a209f00ea (DEFAULT_TABLE table=rebalance_test_table_2 [id=a8027b31a8b34320800f512d75105639]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:55.665354 6424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d63a75f2194484f8a59709a209f00ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:55.665371 6687 tablet_service.cc:1467] Processing CreateTablet for tablet 460b4d38c63a4a4ca177a37be70d927f (DEFAULT_TABLE table=rebalance_test_table_2 [id=a8027b31a8b34320800f512d75105639]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:55.665638 6687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 460b4d38c63a4a4ca177a37be70d927f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:55.665663 6295 tablet_service.cc:1467] Processing CreateTablet for tablet 460b4d38c63a4a4ca177a37be70d927f (DEFAULT_TABLE table=rebalance_test_table_2 [id=a8027b31a8b34320800f512d75105639]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:55.665921 6295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 460b4d38c63a4a4ca177a37be70d927f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:55.667690 7451 tablet_bootstrap.cc:492] T 7d63a75f2194484f8a59709a209f00ea P 8315d22c56f7439a9e1fc82f153f018c: Bootstrap starting.
I20240501 07:58:55.667699 7452 tablet_bootstrap.cc:492] T 460b4d38c63a4a4ca177a37be70d927f P 1f9cb72be98941b5a8e504cbe34132b9: Bootstrap starting.
I20240501 07:58:55.667870 6688 tablet_service.cc:1467] Processing CreateTablet for tablet 7d63a75f2194484f8a59709a209f00ea (DEFAULT_TABLE table=rebalance_test_table_2 [id=a8027b31a8b34320800f512d75105639]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:55.668067 6688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d63a75f2194484f8a59709a209f00ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:55.668537 7451 tablet_bootstrap.cc:654] T 7d63a75f2194484f8a59709a209f00ea P 8315d22c56f7439a9e1fc82f153f018c: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:55.668630 7452 tablet_bootstrap.cc:654] T 460b4d38c63a4a4ca177a37be70d927f P 1f9cb72be98941b5a8e504cbe34132b9: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:55.668561 6557 tablet_service.cc:1467] Processing CreateTablet for tablet 7d63a75f2194484f8a59709a209f00ea (DEFAULT_TABLE table=rebalance_test_table_2 [id=a8027b31a8b34320800f512d75105639]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:55.668834 6557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d63a75f2194484f8a59709a209f00ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:55.669783 7452 tablet_bootstrap.cc:492] T 460b4d38c63a4a4ca177a37be70d927f P 1f9cb72be98941b5a8e504cbe34132b9: No bootstrap required, opened a new log
I20240501 07:58:55.669865 7452 ts_tablet_manager.cc:1367] T 460b4d38c63a4a4ca177a37be70d927f P 1f9cb72be98941b5a8e504cbe34132b9: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 07:58:55.670323 7452 raft_consensus.cc:348] T 460b4d38c63a4a4ca177a37be70d927f P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:58:55.670431 7452 raft_consensus.cc:374] T 460b4d38c63a4a4ca177a37be70d927f P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:55.670464 7452 raft_consensus.cc:729] T 460b4d38c63a4a4ca177a37be70d927f P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f9cb72be98941b5a8e504cbe34132b9, State: Initialized, Role: FOLLOWER
I20240501 07:58:55.670536 6424 tablet_service.cc:1467] Processing CreateTablet for tablet c55dadffc24340338d0439b0ab2855f4 (DEFAULT_TABLE table=rebalance_test_table_2 [id=a8027b31a8b34320800f512d75105639]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:55.670632 7452 consensus_queue.cc:260] T 460b4d38c63a4a4ca177a37be70d927f P 1f9cb72be98941b5a8e504cbe34132b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:58:55.670770 6424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c55dadffc24340338d0439b0ab2855f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:55.670966 6688 tablet_service.cc:1467] Processing CreateTablet for tablet c55dadffc24340338d0439b0ab2855f4 (DEFAULT_TABLE table=rebalance_test_table_2 [id=a8027b31a8b34320800f512d75105639]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:55.671155 6688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c55dadffc24340338d0439b0ab2855f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:55.671463 7452 ts_tablet_manager.cc:1398] T 460b4d38c63a4a4ca177a37be70d927f P 1f9cb72be98941b5a8e504cbe34132b9: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:58:55.671619 7452 tablet_bootstrap.cc:492] T 7d63a75f2194484f8a59709a209f00ea P 1f9cb72be98941b5a8e504cbe34132b9: Bootstrap starting.
I20240501 07:58:55.671754 6295 tablet_service.cc:1467] Processing CreateTablet for tablet c55dadffc24340338d0439b0ab2855f4 (DEFAULT_TABLE table=rebalance_test_table_2 [id=a8027b31a8b34320800f512d75105639]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:55.671998 6295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c55dadffc24340338d0439b0ab2855f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:55.672719 7452 tablet_bootstrap.cc:654] T 7d63a75f2194484f8a59709a209f00ea P 1f9cb72be98941b5a8e504cbe34132b9: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:55.673314 7453 tablet_bootstrap.cc:492] T 460b4d38c63a4a4ca177a37be70d927f P 8b8b3ad3aa8f4053b2167dd550f2086f: Bootstrap starting.
I20240501 07:58:55.674183 7453 tablet_bootstrap.cc:654] T 460b4d38c63a4a4ca177a37be70d927f P 8b8b3ad3aa8f4053b2167dd550f2086f: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:55.674263 7456 tablet_bootstrap.cc:492] T 7d63a75f2194484f8a59709a209f00ea P 3b164cbbc00f4e8eb3254ddd349a3ff9: Bootstrap starting.
I20240501 07:58:55.675022 7456 tablet_bootstrap.cc:654] T 7d63a75f2194484f8a59709a209f00ea P 3b164cbbc00f4e8eb3254ddd349a3ff9: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:55.675251 7452 tablet_bootstrap.cc:492] T 7d63a75f2194484f8a59709a209f00ea P 1f9cb72be98941b5a8e504cbe34132b9: No bootstrap required, opened a new log
I20240501 07:58:55.675330 7452 ts_tablet_manager.cc:1367] T 7d63a75f2194484f8a59709a209f00ea P 1f9cb72be98941b5a8e504cbe34132b9: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 07:58:55.675750 7452 raft_consensus.cc:348] T 7d63a75f2194484f8a59709a209f00ea P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } }
I20240501 07:58:55.675861 7452 raft_consensus.cc:374] T 7d63a75f2194484f8a59709a209f00ea P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:55.675901 7452 raft_consensus.cc:729] T 7d63a75f2194484f8a59709a209f00ea P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f9cb72be98941b5a8e504cbe34132b9, State: Initialized, Role: FOLLOWER
I20240501 07:58:55.676045 7452 consensus_queue.cc:260] T 7d63a75f2194484f8a59709a209f00ea P 1f9cb72be98941b5a8e504cbe34132b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } }
I20240501 07:58:55.676119 7456 tablet_bootstrap.cc:492] T 7d63a75f2194484f8a59709a209f00ea P 3b164cbbc00f4e8eb3254ddd349a3ff9: No bootstrap required, opened a new log
I20240501 07:58:55.676206 7456 ts_tablet_manager.cc:1367] T 7d63a75f2194484f8a59709a209f00ea P 3b164cbbc00f4e8eb3254ddd349a3ff9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:58:55.676311 7452 ts_tablet_manager.cc:1398] T 7d63a75f2194484f8a59709a209f00ea P 1f9cb72be98941b5a8e504cbe34132b9: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:58:55.676666 7456 raft_consensus.cc:348] T 7d63a75f2194484f8a59709a209f00ea P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } }
I20240501 07:58:55.676728 7453 tablet_bootstrap.cc:492] T 460b4d38c63a4a4ca177a37be70d927f P 8b8b3ad3aa8f4053b2167dd550f2086f: No bootstrap required, opened a new log
I20240501 07:58:55.676772 7456 raft_consensus.cc:374] T 7d63a75f2194484f8a59709a209f00ea P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:55.676803 7456 raft_consensus.cc:729] T 7d63a75f2194484f8a59709a209f00ea P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b164cbbc00f4e8eb3254ddd349a3ff9, State: Initialized, Role: FOLLOWER
I20240501 07:58:55.676816 7453 ts_tablet_manager.cc:1367] T 460b4d38c63a4a4ca177a37be70d927f P 8b8b3ad3aa8f4053b2167dd550f2086f: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240501 07:58:55.676908 7456 consensus_queue.cc:260] T 7d63a75f2194484f8a59709a209f00ea P 3b164cbbc00f4e8eb3254ddd349a3ff9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } }
I20240501 07:58:55.677316 7453 raft_consensus.cc:348] T 460b4d38c63a4a4ca177a37be70d927f P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:58:55.677405 7453 raft_consensus.cc:374] T 460b4d38c63a4a4ca177a37be70d927f P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:55.677439 7453 raft_consensus.cc:729] T 460b4d38c63a4a4ca177a37be70d927f P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b8b3ad3aa8f4053b2167dd550f2086f, State: Initialized, Role: FOLLOWER
I20240501 07:58:55.677578 7453 consensus_queue.cc:260] T 460b4d38c63a4a4ca177a37be70d927f P 8b8b3ad3aa8f4053b2167dd550f2086f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:58:55.677690 7456 ts_tablet_manager.cc:1398] T 7d63a75f2194484f8a59709a209f00ea P 3b164cbbc00f4e8eb3254ddd349a3ff9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:55.677907 7453 ts_tablet_manager.cc:1398] T 460b4d38c63a4a4ca177a37be70d927f P 8b8b3ad3aa8f4053b2167dd550f2086f: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 07:58:55.678119 7452 tablet_bootstrap.cc:492] T c55dadffc24340338d0439b0ab2855f4 P 1f9cb72be98941b5a8e504cbe34132b9: Bootstrap starting.
I20240501 07:58:55.678820 7451 tablet_bootstrap.cc:492] T 7d63a75f2194484f8a59709a209f00ea P 8315d22c56f7439a9e1fc82f153f018c: No bootstrap required, opened a new log
I20240501 07:58:55.678918 7451 ts_tablet_manager.cc:1367] T 7d63a75f2194484f8a59709a209f00ea P 8315d22c56f7439a9e1fc82f153f018c: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240501 07:58:55.679015 7452 tablet_bootstrap.cc:654] T c55dadffc24340338d0439b0ab2855f4 P 1f9cb72be98941b5a8e504cbe34132b9: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:55.679390 7451 raft_consensus.cc:348] T 7d63a75f2194484f8a59709a209f00ea P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } }
I20240501 07:58:55.679507 7451 raft_consensus.cc:374] T 7d63a75f2194484f8a59709a209f00ea P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:55.679540 7451 raft_consensus.cc:729] T 7d63a75f2194484f8a59709a209f00ea P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8315d22c56f7439a9e1fc82f153f018c, State: Initialized, Role: FOLLOWER
I20240501 07:58:55.679646 7453 tablet_bootstrap.cc:492] T c55dadffc24340338d0439b0ab2855f4 P 8b8b3ad3aa8f4053b2167dd550f2086f: Bootstrap starting.
I20240501 07:58:55.679652 7451 consensus_queue.cc:260] T 7d63a75f2194484f8a59709a209f00ea P 8315d22c56f7439a9e1fc82f153f018c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } }
I20240501 07:58:55.679947 7451 ts_tablet_manager.cc:1398] T 7d63a75f2194484f8a59709a209f00ea P 8315d22c56f7439a9e1fc82f153f018c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:55.680126 7451 tablet_bootstrap.cc:492] T 460b4d38c63a4a4ca177a37be70d927f P 8315d22c56f7439a9e1fc82f153f018c: Bootstrap starting.
I20240501 07:58:55.680629 7453 tablet_bootstrap.cc:654] T c55dadffc24340338d0439b0ab2855f4 P 8b8b3ad3aa8f4053b2167dd550f2086f: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:55.680639 7452 tablet_bootstrap.cc:492] T c55dadffc24340338d0439b0ab2855f4 P 1f9cb72be98941b5a8e504cbe34132b9: No bootstrap required, opened a new log
I20240501 07:58:55.680707 7452 ts_tablet_manager.cc:1367] T c55dadffc24340338d0439b0ab2855f4 P 1f9cb72be98941b5a8e504cbe34132b9: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240501 07:58:55.681151 7452 raft_consensus.cc:348] T c55dadffc24340338d0439b0ab2855f4 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } }
I20240501 07:58:55.681267 7452 raft_consensus.cc:374] T c55dadffc24340338d0439b0ab2855f4 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:55.681306 7452 raft_consensus.cc:729] T c55dadffc24340338d0439b0ab2855f4 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f9cb72be98941b5a8e504cbe34132b9, State: Initialized, Role: FOLLOWER
I20240501 07:58:55.681432 7452 consensus_queue.cc:260] T c55dadffc24340338d0439b0ab2855f4 P 1f9cb72be98941b5a8e504cbe34132b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } }
I20240501 07:58:55.681684 7453 tablet_bootstrap.cc:492] T c55dadffc24340338d0439b0ab2855f4 P 8b8b3ad3aa8f4053b2167dd550f2086f: No bootstrap required, opened a new log
I20240501 07:58:55.681697 7452 ts_tablet_manager.cc:1398] T c55dadffc24340338d0439b0ab2855f4 P 1f9cb72be98941b5a8e504cbe34132b9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:55.681792 7453 ts_tablet_manager.cc:1367] T c55dadffc24340338d0439b0ab2855f4 P 8b8b3ad3aa8f4053b2167dd550f2086f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240501 07:58:55.682261 7451 tablet_bootstrap.cc:654] T 460b4d38c63a4a4ca177a37be70d927f P 8315d22c56f7439a9e1fc82f153f018c: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:55.682240 7453 raft_consensus.cc:348] T c55dadffc24340338d0439b0ab2855f4 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } }
I20240501 07:58:55.682330 7453 raft_consensus.cc:374] T c55dadffc24340338d0439b0ab2855f4 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:55.682361 7453 raft_consensus.cc:729] T c55dadffc24340338d0439b0ab2855f4 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b8b3ad3aa8f4053b2167dd550f2086f, State: Initialized, Role: FOLLOWER
I20240501 07:58:55.682495 7453 consensus_queue.cc:260] T c55dadffc24340338d0439b0ab2855f4 P 8b8b3ad3aa8f4053b2167dd550f2086f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } }
I20240501 07:58:55.682767 7453 ts_tablet_manager.cc:1398] T c55dadffc24340338d0439b0ab2855f4 P 8b8b3ad3aa8f4053b2167dd550f2086f: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240501 07:58:55.684501 7451 tablet_bootstrap.cc:492] T 460b4d38c63a4a4ca177a37be70d927f P 8315d22c56f7439a9e1fc82f153f018c: No bootstrap required, opened a new log
I20240501 07:58:55.684587 7451 ts_tablet_manager.cc:1367] T 460b4d38c63a4a4ca177a37be70d927f P 8315d22c56f7439a9e1fc82f153f018c: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240501 07:58:55.685225 7451 raft_consensus.cc:348] T 460b4d38c63a4a4ca177a37be70d927f P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:58:55.685323 7451 raft_consensus.cc:374] T 460b4d38c63a4a4ca177a37be70d927f P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:55.685375 7451 raft_consensus.cc:729] T 460b4d38c63a4a4ca177a37be70d927f P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8315d22c56f7439a9e1fc82f153f018c, State: Initialized, Role: FOLLOWER
I20240501 07:58:55.685479 7451 consensus_queue.cc:260] T 460b4d38c63a4a4ca177a37be70d927f P 8315d22c56f7439a9e1fc82f153f018c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:58:55.685721 7451 ts_tablet_manager.cc:1398] T 460b4d38c63a4a4ca177a37be70d927f P 8315d22c56f7439a9e1fc82f153f018c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:58:55.685920 7451 tablet_bootstrap.cc:492] T c55dadffc24340338d0439b0ab2855f4 P 8315d22c56f7439a9e1fc82f153f018c: Bootstrap starting.
I20240501 07:58:55.686779 7451 tablet_bootstrap.cc:654] T c55dadffc24340338d0439b0ab2855f4 P 8315d22c56f7439a9e1fc82f153f018c: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:55.687732 7451 tablet_bootstrap.cc:492] T c55dadffc24340338d0439b0ab2855f4 P 8315d22c56f7439a9e1fc82f153f018c: No bootstrap required, opened a new log
I20240501 07:58:55.687803 7451 ts_tablet_manager.cc:1367] T c55dadffc24340338d0439b0ab2855f4 P 8315d22c56f7439a9e1fc82f153f018c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:58:55.688253 7451 raft_consensus.cc:348] T c55dadffc24340338d0439b0ab2855f4 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } }
I20240501 07:58:55.688331 7451 raft_consensus.cc:374] T c55dadffc24340338d0439b0ab2855f4 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:55.688354 7451 raft_consensus.cc:729] T c55dadffc24340338d0439b0ab2855f4 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8315d22c56f7439a9e1fc82f153f018c, State: Initialized, Role: FOLLOWER
I20240501 07:58:55.688464 7451 consensus_queue.cc:260] T c55dadffc24340338d0439b0ab2855f4 P 8315d22c56f7439a9e1fc82f153f018c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } }
I20240501 07:58:55.688707 7451 ts_tablet_manager.cc:1398] T c55dadffc24340338d0439b0ab2855f4 P 8315d22c56f7439a9e1fc82f153f018c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:55.771970 7306 raft_consensus.cc:482] T c55dadffc24340338d0439b0ab2855f4 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:55.772107 7306 raft_consensus.cc:504] T c55dadffc24340338d0439b0ab2855f4 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } }
I20240501 07:58:55.772521 7306 leader_election.cc:288] T c55dadffc24340338d0439b0ab2855f4 P 8b8b3ad3aa8f4053b2167dd550f2086f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130:32941), 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132:36773)
I20240501 07:58:55.772974 6446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c55dadffc24340338d0439b0ab2855f4" candidate_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8315d22c56f7439a9e1fc82f153f018c" is_pre_election: true
I20240501 07:58:55.773069 6708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c55dadffc24340338d0439b0ab2855f4" candidate_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f9cb72be98941b5a8e504cbe34132b9" is_pre_election: true
I20240501 07:58:55.773135 6446 raft_consensus.cc:2454] T c55dadffc24340338d0439b0ab2855f4 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b8b3ad3aa8f4053b2167dd550f2086f in term 0.
I20240501 07:58:55.773219 6708 raft_consensus.cc:2454] T c55dadffc24340338d0439b0ab2855f4 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b8b3ad3aa8f4053b2167dd550f2086f in term 0.
I20240501 07:58:55.773500 6250 leader_election.cc:302] T c55dadffc24340338d0439b0ab2855f4 P 8b8b3ad3aa8f4053b2167dd550f2086f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8315d22c56f7439a9e1fc82f153f018c, 8b8b3ad3aa8f4053b2167dd550f2086f; no voters:
I20240501 07:58:55.773696 7306 raft_consensus.cc:2781] T c55dadffc24340338d0439b0ab2855f4 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:55.773766 7306 raft_consensus.cc:482] T c55dadffc24340338d0439b0ab2855f4 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:55.773794 7306 raft_consensus.cc:3037] T c55dadffc24340338d0439b0ab2855f4 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:55.774648 7306 raft_consensus.cc:504] T c55dadffc24340338d0439b0ab2855f4 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } }
I20240501 07:58:55.774966 7306 leader_election.cc:288] T c55dadffc24340338d0439b0ab2855f4 P 8b8b3ad3aa8f4053b2167dd550f2086f [CANDIDATE]: Term 1 election: Requested vote from peers 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130:32941), 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132:36773)
I20240501 07:58:55.775279 6446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c55dadffc24340338d0439b0ab2855f4" candidate_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8315d22c56f7439a9e1fc82f153f018c"
I20240501 07:58:55.775380 6446 raft_consensus.cc:3037] T c55dadffc24340338d0439b0ab2855f4 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:55.775343 6708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c55dadffc24340338d0439b0ab2855f4" candidate_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f9cb72be98941b5a8e504cbe34132b9"
I20240501 07:58:55.775475 6708 raft_consensus.cc:3037] T c55dadffc24340338d0439b0ab2855f4 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:55.776144 6446 raft_consensus.cc:2454] T c55dadffc24340338d0439b0ab2855f4 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b8b3ad3aa8f4053b2167dd550f2086f in term 1.
I20240501 07:58:55.776268 6708 raft_consensus.cc:2454] T c55dadffc24340338d0439b0ab2855f4 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b8b3ad3aa8f4053b2167dd550f2086f in term 1.
I20240501 07:58:55.776408 6250 leader_election.cc:302] T c55dadffc24340338d0439b0ab2855f4 P 8b8b3ad3aa8f4053b2167dd550f2086f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8315d22c56f7439a9e1fc82f153f018c, 8b8b3ad3aa8f4053b2167dd550f2086f; no voters:
I20240501 07:58:55.776563 7306 raft_consensus.cc:2781] T c55dadffc24340338d0439b0ab2855f4 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:55.776758 7306 raft_consensus.cc:686] T c55dadffc24340338d0439b0ab2855f4 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 LEADER]: Becoming Leader. State: Replica: 8b8b3ad3aa8f4053b2167dd550f2086f, State: Running, Role: LEADER
I20240501 07:58:55.776897 7306 consensus_queue.cc:237] T c55dadffc24340338d0439b0ab2855f4 P 8b8b3ad3aa8f4053b2167dd550f2086f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } }
I20240501 07:58:55.778815 6170 catalog_manager.cc:5484] T c55dadffc24340338d0439b0ab2855f4 P 8b8b3ad3aa8f4053b2167dd550f2086f reported cstate change: term changed from 0 to 1, leader changed from <none> to 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129). New cstate: current_term: 1 leader_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } health_report { overall_health: HEALTHY } } }
I20240501 07:58:55.798370 7313 raft_consensus.cc:482] T 460b4d38c63a4a4ca177a37be70d927f P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:55.798506 7313 raft_consensus.cc:504] T 460b4d38c63a4a4ca177a37be70d927f P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:58:55.799038 7313 leader_election.cc:288] T 460b4d38c63a4a4ca177a37be70d927f P 1f9cb72be98941b5a8e504cbe34132b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130:32941), 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129:46875)
I20240501 07:58:55.799512 6314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "460b4d38c63a4a4ca177a37be70d927f" candidate_uuid: "1f9cb72be98941b5a8e504cbe34132b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" is_pre_election: true
I20240501 07:58:55.799688 6314 raft_consensus.cc:2454] T 460b4d38c63a4a4ca177a37be70d927f P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f9cb72be98941b5a8e504cbe34132b9 in term 0.
I20240501 07:58:55.800021 6641 leader_election.cc:302] T 460b4d38c63a4a4ca177a37be70d927f P 1f9cb72be98941b5a8e504cbe34132b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f9cb72be98941b5a8e504cbe34132b9, 8b8b3ad3aa8f4053b2167dd550f2086f; no voters:
I20240501 07:58:55.800181 7313 raft_consensus.cc:2781] T 460b4d38c63a4a4ca177a37be70d927f P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:55.800235 7313 raft_consensus.cc:482] T 460b4d38c63a4a4ca177a37be70d927f P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:55.800289 7313 raft_consensus.cc:3037] T 460b4d38c63a4a4ca177a37be70d927f P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:55.801213 7313 raft_consensus.cc:504] T 460b4d38c63a4a4ca177a37be70d927f P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:58:55.801550 7313 leader_election.cc:288] T 460b4d38c63a4a4ca177a37be70d927f P 1f9cb72be98941b5a8e504cbe34132b9 [CANDIDATE]: Term 1 election: Requested vote from peers 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130:32941), 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129:46875)
I20240501 07:58:55.801950 6314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "460b4d38c63a4a4ca177a37be70d927f" candidate_uuid: "1f9cb72be98941b5a8e504cbe34132b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f"
I20240501 07:58:55.802073 6314 raft_consensus.cc:3037] T 460b4d38c63a4a4ca177a37be70d927f P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:55.802819 6446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "460b4d38c63a4a4ca177a37be70d927f" candidate_uuid: "1f9cb72be98941b5a8e504cbe34132b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8315d22c56f7439a9e1fc82f153f018c" is_pre_election: true
I20240501 07:58:55.802826 6445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "460b4d38c63a4a4ca177a37be70d927f" candidate_uuid: "1f9cb72be98941b5a8e504cbe34132b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8315d22c56f7439a9e1fc82f153f018c"
I20240501 07:58:55.802898 6314 raft_consensus.cc:2454] T 460b4d38c63a4a4ca177a37be70d927f P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f9cb72be98941b5a8e504cbe34132b9 in term 1.
I20240501 07:58:55.802976 6446 raft_consensus.cc:2454] T 460b4d38c63a4a4ca177a37be70d927f P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f9cb72be98941b5a8e504cbe34132b9 in term 0.
I20240501 07:58:55.803210 6641 leader_election.cc:302] T 460b4d38c63a4a4ca177a37be70d927f P 1f9cb72be98941b5a8e504cbe34132b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f9cb72be98941b5a8e504cbe34132b9, 8b8b3ad3aa8f4053b2167dd550f2086f; no voters:
I20240501 07:58:55.803370 7313 raft_consensus.cc:2781] T 460b4d38c63a4a4ca177a37be70d927f P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:55.803570 7313 raft_consensus.cc:686] T 460b4d38c63a4a4ca177a37be70d927f P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 LEADER]: Becoming Leader. State: Replica: 1f9cb72be98941b5a8e504cbe34132b9, State: Running, Role: LEADER
I20240501 07:58:55.803859 7313 consensus_queue.cc:237] T 460b4d38c63a4a4ca177a37be70d927f P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:58:55.806226 6170 catalog_manager.cc:5484] T 460b4d38c63a4a4ca177a37be70d927f P 1f9cb72be98941b5a8e504cbe34132b9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132). New cstate: current_term: 1 leader_uuid: "1f9cb72be98941b5a8e504cbe34132b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } health_report { overall_health: HEALTHY } } }
I20240501 07:58:55.889364 7433 raft_consensus.cc:482] T 7d63a75f2194484f8a59709a209f00ea P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:55.889516 7433 raft_consensus.cc:504] T 7d63a75f2194484f8a59709a209f00ea P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } }
I20240501 07:58:55.890694 7433 leader_election.cc:288] T 7d63a75f2194484f8a59709a209f00ea P 8315d22c56f7439a9e1fc82f153f018c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132:36773), 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131:34471)
I20240501 07:58:55.894286 6708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d63a75f2194484f8a59709a209f00ea" candidate_uuid: "8315d22c56f7439a9e1fc82f153f018c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f9cb72be98941b5a8e504cbe34132b9" is_pre_election: true
I20240501 07:58:55.894342 6577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d63a75f2194484f8a59709a209f00ea" candidate_uuid: "8315d22c56f7439a9e1fc82f153f018c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" is_pre_election: true
I20240501 07:58:55.894431 6708 raft_consensus.cc:2454] T 7d63a75f2194484f8a59709a209f00ea P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8315d22c56f7439a9e1fc82f153f018c in term 0.
I20240501 07:58:55.894539 6577 raft_consensus.cc:2454] T 7d63a75f2194484f8a59709a209f00ea P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8315d22c56f7439a9e1fc82f153f018c in term 0.
I20240501 07:58:55.894773 6381 leader_election.cc:302] T 7d63a75f2194484f8a59709a209f00ea P 8315d22c56f7439a9e1fc82f153f018c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f9cb72be98941b5a8e504cbe34132b9, 8315d22c56f7439a9e1fc82f153f018c; no voters:
I20240501 07:58:55.895010 7433 raft_consensus.cc:2781] T 7d63a75f2194484f8a59709a209f00ea P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:55.895067 7433 raft_consensus.cc:482] T 7d63a75f2194484f8a59709a209f00ea P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:55.895094 7433 raft_consensus.cc:3037] T 7d63a75f2194484f8a59709a209f00ea P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:55.895960 7433 raft_consensus.cc:504] T 7d63a75f2194484f8a59709a209f00ea P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } }
I20240501 07:58:55.896304 7433 leader_election.cc:288] T 7d63a75f2194484f8a59709a209f00ea P 8315d22c56f7439a9e1fc82f153f018c [CANDIDATE]: Term 1 election: Requested vote from peers 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132:36773), 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131:34471)
I20240501 07:58:55.896626 6708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d63a75f2194484f8a59709a209f00ea" candidate_uuid: "8315d22c56f7439a9e1fc82f153f018c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f9cb72be98941b5a8e504cbe34132b9"
I20240501 07:58:55.896729 6708 raft_consensus.cc:3037] T 7d63a75f2194484f8a59709a209f00ea P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:55.897087 6577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d63a75f2194484f8a59709a209f00ea" candidate_uuid: "8315d22c56f7439a9e1fc82f153f018c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9"
I20240501 07:58:55.897215 6577 raft_consensus.cc:3037] T 7d63a75f2194484f8a59709a209f00ea P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:55.897692 6708 raft_consensus.cc:2454] T 7d63a75f2194484f8a59709a209f00ea P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8315d22c56f7439a9e1fc82f153f018c in term 1.
I20240501 07:58:55.897977 6381 leader_election.cc:302] T 7d63a75f2194484f8a59709a209f00ea P 8315d22c56f7439a9e1fc82f153f018c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f9cb72be98941b5a8e504cbe34132b9, 8315d22c56f7439a9e1fc82f153f018c; no voters:
I20240501 07:58:55.898156 7433 raft_consensus.cc:2781] T 7d63a75f2194484f8a59709a209f00ea P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:55.898394 7433 raft_consensus.cc:686] T 7d63a75f2194484f8a59709a209f00ea P 8315d22c56f7439a9e1fc82f153f018c [term 1 LEADER]: Becoming Leader. State: Replica: 8315d22c56f7439a9e1fc82f153f018c, State: Running, Role: LEADER
I20240501 07:58:55.898396 6577 raft_consensus.cc:2454] T 7d63a75f2194484f8a59709a209f00ea P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8315d22c56f7439a9e1fc82f153f018c in term 1.
I20240501 07:58:55.898741 7433 consensus_queue.cc:237] T 7d63a75f2194484f8a59709a209f00ea P 8315d22c56f7439a9e1fc82f153f018c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } }
I20240501 07:58:55.900857 6170 catalog_manager.cc:5484] T 7d63a75f2194484f8a59709a209f00ea P 8315d22c56f7439a9e1fc82f153f018c reported cstate change: term changed from 0 to 1, leader changed from <none> to 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130). New cstate: current_term: 1 leader_uuid: "8315d22c56f7439a9e1fc82f153f018c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } health_report { overall_health: UNKNOWN } } }
W20240501 07:58:56.014972 7474 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:58:56.015252 7474 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:58:56.019860 7474 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
I20240501 07:58:56.110491 6708 raft_consensus.cc:1261] T c55dadffc24340338d0439b0ab2855f4 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Refusing update from remote peer 8b8b3ad3aa8f4053b2167dd550f2086f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:56.110549 6446 raft_consensus.cc:1261] T c55dadffc24340338d0439b0ab2855f4 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Refusing update from remote peer 8b8b3ad3aa8f4053b2167dd550f2086f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:56.110592 6577 raft_consensus.cc:1261] T 7d63a75f2194484f8a59709a209f00ea P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Refusing update from remote peer 8315d22c56f7439a9e1fc82f153f018c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:56.111040 7306 consensus_queue.cc:1035] T c55dadffc24340338d0439b0ab2855f4 P 8b8b3ad3aa8f4053b2167dd550f2086f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:56.111336 7464 consensus_queue.cc:1035] T c55dadffc24340338d0439b0ab2855f4 P 8b8b3ad3aa8f4053b2167dd550f2086f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:56.112203 7433 consensus_queue.cc:1035] T 7d63a75f2194484f8a59709a209f00ea P 8315d22c56f7439a9e1fc82f153f018c [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:56.115088 6707 raft_consensus.cc:1261] T 7d63a75f2194484f8a59709a209f00ea P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Refusing update from remote peer 8315d22c56f7439a9e1fc82f153f018c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:56.115540 7472 consensus_queue.cc:1035] T 7d63a75f2194484f8a59709a209f00ea P 8315d22c56f7439a9e1fc82f153f018c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:56.135890 6446 raft_consensus.cc:3037] T 460b4d38c63a4a4ca177a37be70d927f P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:56.137046 6446 raft_consensus.cc:1261] T 460b4d38c63a4a4ca177a37be70d927f P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Refusing update from remote peer 1f9cb72be98941b5a8e504cbe34132b9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:56.137490 7313 consensus_queue.cc:1035] T 460b4d38c63a4a4ca177a37be70d927f P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:56.139029 6314 raft_consensus.cc:1261] T 460b4d38c63a4a4ca177a37be70d927f P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Refusing update from remote peer 1f9cb72be98941b5a8e504cbe34132b9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:56.139452 7313 consensus_queue.cc:1035] T 460b4d38c63a4a4ca177a37be70d927f P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 123.452 ms
time per row: 0.0617259 ms
I20240501 07:58:56.188853 6173 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55950:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 07:58:56.194371 6424 tablet_service.cc:1467] Processing CreateTablet for tablet 782e37a4ebb34a68b461a74115ae8703 (DEFAULT_TABLE table=rebalance_test_table_3 [id=eb6765f7643442ff82fde7f790b2aaed]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:56.194726 6424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 782e37a4ebb34a68b461a74115ae8703. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:56.194823 6556 tablet_service.cc:1467] Processing CreateTablet for tablet 5ee738571af740c1b494d4f17413d1d8 (DEFAULT_TABLE table=rebalance_test_table_3 [id=eb6765f7643442ff82fde7f790b2aaed]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:56.195101 6556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ee738571af740c1b494d4f17413d1d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:56.195547 6294 tablet_service.cc:1467] Processing CreateTablet for tablet 5ee738571af740c1b494d4f17413d1d8 (DEFAULT_TABLE table=rebalance_test_table_3 [id=eb6765f7643442ff82fde7f790b2aaed]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:56.195808 6294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ee738571af740c1b494d4f17413d1d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:56.196830 7521 tablet_bootstrap.cc:492] T 782e37a4ebb34a68b461a74115ae8703 P 8315d22c56f7439a9e1fc82f153f018c: Bootstrap starting.
I20240501 07:58:56.197053 6557 tablet_service.cc:1467] Processing CreateTablet for tablet 782e37a4ebb34a68b461a74115ae8703 (DEFAULT_TABLE table=rebalance_test_table_3 [id=eb6765f7643442ff82fde7f790b2aaed]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:56.197306 6557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 782e37a4ebb34a68b461a74115ae8703. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:56.198058 7521 tablet_bootstrap.cc:654] T 782e37a4ebb34a68b461a74115ae8703 P 8315d22c56f7439a9e1fc82f153f018c: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:56.198686 7522 tablet_bootstrap.cc:492] T 5ee738571af740c1b494d4f17413d1d8 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Bootstrap starting.
I20240501 07:58:56.198946 6688 tablet_service.cc:1467] Processing CreateTablet for tablet 5ee738571af740c1b494d4f17413d1d8 (DEFAULT_TABLE table=rebalance_test_table_3 [id=eb6765f7643442ff82fde7f790b2aaed]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:56.199218 6688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ee738571af740c1b494d4f17413d1d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:56.199262 7521 tablet_bootstrap.cc:492] T 782e37a4ebb34a68b461a74115ae8703 P 8315d22c56f7439a9e1fc82f153f018c: No bootstrap required, opened a new log
I20240501 07:58:56.199342 7521 ts_tablet_manager.cc:1367] T 782e37a4ebb34a68b461a74115ae8703 P 8315d22c56f7439a9e1fc82f153f018c: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 07:58:56.199527 6295 tablet_service.cc:1467] Processing CreateTablet for tablet 782e37a4ebb34a68b461a74115ae8703 (DEFAULT_TABLE table=rebalance_test_table_3 [id=eb6765f7643442ff82fde7f790b2aaed]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:56.199656 7522 tablet_bootstrap.cc:654] T 5ee738571af740c1b494d4f17413d1d8 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:56.199757 6295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 782e37a4ebb34a68b461a74115ae8703. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:56.199827 7521 raft_consensus.cc:348] T 782e37a4ebb34a68b461a74115ae8703 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } }
I20240501 07:58:56.199947 7521 raft_consensus.cc:374] T 782e37a4ebb34a68b461a74115ae8703 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:56.199980 7521 raft_consensus.cc:729] T 782e37a4ebb34a68b461a74115ae8703 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8315d22c56f7439a9e1fc82f153f018c, State: Initialized, Role: FOLLOWER
I20240501 07:58:56.200119 7521 consensus_queue.cc:260] T 782e37a4ebb34a68b461a74115ae8703 P 8315d22c56f7439a9e1fc82f153f018c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } }
I20240501 07:58:56.200399 7521 ts_tablet_manager.cc:1398] T 782e37a4ebb34a68b461a74115ae8703 P 8315d22c56f7439a9e1fc82f153f018c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:56.201318 6687 tablet_service.cc:1467] Processing CreateTablet for tablet 8663a511521644009fc360ad92c093a1 (DEFAULT_TABLE table=rebalance_test_table_3 [id=eb6765f7643442ff82fde7f790b2aaed]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:56.201581 6687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8663a511521644009fc360ad92c093a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:56.202245 7523 tablet_bootstrap.cc:492] T 5ee738571af740c1b494d4f17413d1d8 P 8b8b3ad3aa8f4053b2167dd550f2086f: Bootstrap starting.
I20240501 07:58:56.202317 6424 tablet_service.cc:1467] Processing CreateTablet for tablet 8663a511521644009fc360ad92c093a1 (DEFAULT_TABLE table=rebalance_test_table_3 [id=eb6765f7643442ff82fde7f790b2aaed]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:56.202526 6424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8663a511521644009fc360ad92c093a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:56.203120 7523 tablet_bootstrap.cc:654] T 5ee738571af740c1b494d4f17413d1d8 P 8b8b3ad3aa8f4053b2167dd550f2086f: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:56.203780 7526 tablet_bootstrap.cc:492] T 5ee738571af740c1b494d4f17413d1d8 P 1f9cb72be98941b5a8e504cbe34132b9: Bootstrap starting.
I20240501 07:58:56.204135 6557 tablet_service.cc:1467] Processing CreateTablet for tablet 8663a511521644009fc360ad92c093a1 (DEFAULT_TABLE table=rebalance_test_table_3 [id=eb6765f7643442ff82fde7f790b2aaed]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:56.204346 6557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8663a511521644009fc360ad92c093a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:56.204596 7526 tablet_bootstrap.cc:654] T 5ee738571af740c1b494d4f17413d1d8 P 1f9cb72be98941b5a8e504cbe34132b9: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:56.205224 7521 tablet_bootstrap.cc:492] T 8663a511521644009fc360ad92c093a1 P 8315d22c56f7439a9e1fc82f153f018c: Bootstrap starting.
I20240501 07:58:56.208315 7522 tablet_bootstrap.cc:492] T 5ee738571af740c1b494d4f17413d1d8 P 3b164cbbc00f4e8eb3254ddd349a3ff9: No bootstrap required, opened a new log
I20240501 07:58:56.208403 7522 ts_tablet_manager.cc:1367] T 5ee738571af740c1b494d4f17413d1d8 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.000s
I20240501 07:58:56.208410 7523 tablet_bootstrap.cc:492] T 5ee738571af740c1b494d4f17413d1d8 P 8b8b3ad3aa8f4053b2167dd550f2086f: No bootstrap required, opened a new log
I20240501 07:58:56.208513 7523 ts_tablet_manager.cc:1367] T 5ee738571af740c1b494d4f17413d1d8 P 8b8b3ad3aa8f4053b2167dd550f2086f: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 07:58:56.208797 7522 raft_consensus.cc:348] T 5ee738571af740c1b494d4f17413d1d8 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:58:56.208930 7522 raft_consensus.cc:374] T 5ee738571af740c1b494d4f17413d1d8 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:56.209014 7522 raft_consensus.cc:729] T 5ee738571af740c1b494d4f17413d1d8 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b164cbbc00f4e8eb3254ddd349a3ff9, State: Initialized, Role: FOLLOWER
I20240501 07:58:56.209127 7522 consensus_queue.cc:260] T 5ee738571af740c1b494d4f17413d1d8 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:58:56.209148 7523 raft_consensus.cc:348] T 5ee738571af740c1b494d4f17413d1d8 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:58:56.209246 7523 raft_consensus.cc:374] T 5ee738571af740c1b494d4f17413d1d8 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:56.209275 7523 raft_consensus.cc:729] T 5ee738571af740c1b494d4f17413d1d8 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b8b3ad3aa8f4053b2167dd550f2086f, State: Initialized, Role: FOLLOWER
I20240501 07:58:56.209358 7522 ts_tablet_manager.cc:1398] T 5ee738571af740c1b494d4f17413d1d8 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:56.209499 7522 tablet_bootstrap.cc:492] T 782e37a4ebb34a68b461a74115ae8703 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Bootstrap starting.
I20240501 07:58:56.209483 7523 consensus_queue.cc:260] T 5ee738571af740c1b494d4f17413d1d8 P 8b8b3ad3aa8f4053b2167dd550f2086f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:58:56.209882 7523 ts_tablet_manager.cc:1398] T 5ee738571af740c1b494d4f17413d1d8 P 8b8b3ad3aa8f4053b2167dd550f2086f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:56.210040 7523 tablet_bootstrap.cc:492] T 782e37a4ebb34a68b461a74115ae8703 P 8b8b3ad3aa8f4053b2167dd550f2086f: Bootstrap starting.
I20240501 07:58:56.210240 7522 tablet_bootstrap.cc:654] T 782e37a4ebb34a68b461a74115ae8703 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:56.210289 7521 tablet_bootstrap.cc:654] T 8663a511521644009fc360ad92c093a1 P 8315d22c56f7439a9e1fc82f153f018c: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:56.210959 7523 tablet_bootstrap.cc:654] T 782e37a4ebb34a68b461a74115ae8703 P 8b8b3ad3aa8f4053b2167dd550f2086f: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:56.211956 7523 tablet_bootstrap.cc:492] T 782e37a4ebb34a68b461a74115ae8703 P 8b8b3ad3aa8f4053b2167dd550f2086f: No bootstrap required, opened a new log
I20240501 07:58:56.212033 7523 ts_tablet_manager.cc:1367] T 782e37a4ebb34a68b461a74115ae8703 P 8b8b3ad3aa8f4053b2167dd550f2086f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:58:56.212368 7523 raft_consensus.cc:348] T 782e37a4ebb34a68b461a74115ae8703 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } }
I20240501 07:58:56.212458 7523 raft_consensus.cc:374] T 782e37a4ebb34a68b461a74115ae8703 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:56.212505 7523 raft_consensus.cc:729] T 782e37a4ebb34a68b461a74115ae8703 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b8b3ad3aa8f4053b2167dd550f2086f, State: Initialized, Role: FOLLOWER
I20240501 07:58:56.212651 7523 consensus_queue.cc:260] T 782e37a4ebb34a68b461a74115ae8703 P 8b8b3ad3aa8f4053b2167dd550f2086f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } }
I20240501 07:58:56.212917 7523 ts_tablet_manager.cc:1398] T 782e37a4ebb34a68b461a74115ae8703 P 8b8b3ad3aa8f4053b2167dd550f2086f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:56.213430 7521 tablet_bootstrap.cc:492] T 8663a511521644009fc360ad92c093a1 P 8315d22c56f7439a9e1fc82f153f018c: No bootstrap required, opened a new log
I20240501 07:58:56.213521 7521 ts_tablet_manager.cc:1367] T 8663a511521644009fc360ad92c093a1 P 8315d22c56f7439a9e1fc82f153f018c: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240501 07:58:56.213984 7522 tablet_bootstrap.cc:492] T 782e37a4ebb34a68b461a74115ae8703 P 3b164cbbc00f4e8eb3254ddd349a3ff9: No bootstrap required, opened a new log
I20240501 07:58:56.214046 7522 ts_tablet_manager.cc:1367] T 782e37a4ebb34a68b461a74115ae8703 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240501 07:58:56.214044 7521 raft_consensus.cc:348] T 8663a511521644009fc360ad92c093a1 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:58:56.214149 7521 raft_consensus.cc:374] T 8663a511521644009fc360ad92c093a1 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:56.214180 7521 raft_consensus.cc:729] T 8663a511521644009fc360ad92c093a1 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8315d22c56f7439a9e1fc82f153f018c, State: Initialized, Role: FOLLOWER
I20240501 07:58:56.214305 7521 consensus_queue.cc:260] T 8663a511521644009fc360ad92c093a1 P 8315d22c56f7439a9e1fc82f153f018c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:58:56.214424 7522 raft_consensus.cc:348] T 782e37a4ebb34a68b461a74115ae8703 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } }
I20240501 07:58:56.214511 7522 raft_consensus.cc:374] T 782e37a4ebb34a68b461a74115ae8703 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:56.214536 7522 raft_consensus.cc:729] T 782e37a4ebb34a68b461a74115ae8703 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b164cbbc00f4e8eb3254ddd349a3ff9, State: Initialized, Role: FOLLOWER
I20240501 07:58:56.214560 7521 ts_tablet_manager.cc:1398] T 8663a511521644009fc360ad92c093a1 P 8315d22c56f7439a9e1fc82f153f018c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:56.214638 7522 consensus_queue.cc:260] T 782e37a4ebb34a68b461a74115ae8703 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } }
I20240501 07:58:56.214865 7522 ts_tablet_manager.cc:1398] T 782e37a4ebb34a68b461a74115ae8703 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:56.214994 7522 tablet_bootstrap.cc:492] T 8663a511521644009fc360ad92c093a1 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Bootstrap starting.
I20240501 07:58:56.215778 7522 tablet_bootstrap.cc:654] T 8663a511521644009fc360ad92c093a1 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:56.217659 7522 tablet_bootstrap.cc:492] T 8663a511521644009fc360ad92c093a1 P 3b164cbbc00f4e8eb3254ddd349a3ff9: No bootstrap required, opened a new log
I20240501 07:58:56.217741 7522 ts_tablet_manager.cc:1367] T 8663a511521644009fc360ad92c093a1 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240501 07:58:56.218401 7522 raft_consensus.cc:348] T 8663a511521644009fc360ad92c093a1 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:58:56.218503 7522 raft_consensus.cc:374] T 8663a511521644009fc360ad92c093a1 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:56.218550 7522 raft_consensus.cc:729] T 8663a511521644009fc360ad92c093a1 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b164cbbc00f4e8eb3254ddd349a3ff9, State: Initialized, Role: FOLLOWER
I20240501 07:58:56.218699 7522 consensus_queue.cc:260] T 8663a511521644009fc360ad92c093a1 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:58:56.219080 7522 ts_tablet_manager.cc:1398] T 8663a511521644009fc360ad92c093a1 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:56.219230 7526 tablet_bootstrap.cc:492] T 5ee738571af740c1b494d4f17413d1d8 P 1f9cb72be98941b5a8e504cbe34132b9: No bootstrap required, opened a new log
I20240501 07:58:56.219323 7526 ts_tablet_manager.cc:1367] T 5ee738571af740c1b494d4f17413d1d8 P 1f9cb72be98941b5a8e504cbe34132b9: Time spent bootstrapping tablet: real 0.016s user 0.001s sys 0.000s
I20240501 07:58:56.219745 7526 raft_consensus.cc:348] T 5ee738571af740c1b494d4f17413d1d8 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:58:56.219839 7526 raft_consensus.cc:374] T 5ee738571af740c1b494d4f17413d1d8 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:56.219861 7526 raft_consensus.cc:729] T 5ee738571af740c1b494d4f17413d1d8 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f9cb72be98941b5a8e504cbe34132b9, State: Initialized, Role: FOLLOWER
I20240501 07:58:56.219970 7526 consensus_queue.cc:260] T 5ee738571af740c1b494d4f17413d1d8 P 1f9cb72be98941b5a8e504cbe34132b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:58:56.220216 7526 ts_tablet_manager.cc:1398] T 5ee738571af740c1b494d4f17413d1d8 P 1f9cb72be98941b5a8e504cbe34132b9: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:58:56.220366 7526 tablet_bootstrap.cc:492] T 8663a511521644009fc360ad92c093a1 P 1f9cb72be98941b5a8e504cbe34132b9: Bootstrap starting.
I20240501 07:58:56.221177 7526 tablet_bootstrap.cc:654] T 8663a511521644009fc360ad92c093a1 P 1f9cb72be98941b5a8e504cbe34132b9: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:56.222215 7526 tablet_bootstrap.cc:492] T 8663a511521644009fc360ad92c093a1 P 1f9cb72be98941b5a8e504cbe34132b9: No bootstrap required, opened a new log
I20240501 07:58:56.222296 7526 ts_tablet_manager.cc:1367] T 8663a511521644009fc360ad92c093a1 P 1f9cb72be98941b5a8e504cbe34132b9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:58:56.222764 7526 raft_consensus.cc:348] T 8663a511521644009fc360ad92c093a1 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:58:56.222877 7526 raft_consensus.cc:374] T 8663a511521644009fc360ad92c093a1 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:56.222926 7526 raft_consensus.cc:729] T 8663a511521644009fc360ad92c093a1 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f9cb72be98941b5a8e504cbe34132b9, State: Initialized, Role: FOLLOWER
I20240501 07:58:56.223044 7526 consensus_queue.cc:260] T 8663a511521644009fc360ad92c093a1 P 1f9cb72be98941b5a8e504cbe34132b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:58:56.223278 7526 ts_tablet_manager.cc:1398] T 8663a511521644009fc360ad92c093a1 P 1f9cb72be98941b5a8e504cbe34132b9: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:58:56.226737 7306 raft_consensus.cc:482] T 5ee738571af740c1b494d4f17413d1d8 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:56.226821 7306 raft_consensus.cc:504] T 5ee738571af740c1b494d4f17413d1d8 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:58:56.227205 7306 leader_election.cc:288] T 5ee738571af740c1b494d4f17413d1d8 P 8b8b3ad3aa8f4053b2167dd550f2086f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131:34471), 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132:36773)
I20240501 07:58:56.227588 6708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ee738571af740c1b494d4f17413d1d8" candidate_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f9cb72be98941b5a8e504cbe34132b9" is_pre_election: true
I20240501 07:58:56.227725 6708 raft_consensus.cc:2454] T 5ee738571af740c1b494d4f17413d1d8 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b8b3ad3aa8f4053b2167dd550f2086f in term 0.
I20240501 07:58:56.227977 6250 leader_election.cc:302] T 5ee738571af740c1b494d4f17413d1d8 P 8b8b3ad3aa8f4053b2167dd550f2086f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f9cb72be98941b5a8e504cbe34132b9, 8b8b3ad3aa8f4053b2167dd550f2086f; no voters:
I20240501 07:58:56.228128 7306 raft_consensus.cc:2781] T 5ee738571af740c1b494d4f17413d1d8 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:56.228195 7306 raft_consensus.cc:482] T 5ee738571af740c1b494d4f17413d1d8 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:56.228235 7306 raft_consensus.cc:3037] T 5ee738571af740c1b494d4f17413d1d8 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:56.228936 7306 raft_consensus.cc:504] T 5ee738571af740c1b494d4f17413d1d8 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:58:56.229266 7306 leader_election.cc:288] T 5ee738571af740c1b494d4f17413d1d8 P 8b8b3ad3aa8f4053b2167dd550f2086f [CANDIDATE]: Term 1 election: Requested vote from peers 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131:34471), 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132:36773)
I20240501 07:58:56.229722 6708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ee738571af740c1b494d4f17413d1d8" candidate_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f9cb72be98941b5a8e504cbe34132b9"
I20240501 07:58:56.229827 6708 raft_consensus.cc:3037] T 5ee738571af740c1b494d4f17413d1d8 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:56.230674 6708 raft_consensus.cc:2454] T 5ee738571af740c1b494d4f17413d1d8 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b8b3ad3aa8f4053b2167dd550f2086f in term 1.
I20240501 07:58:56.230931 6250 leader_election.cc:302] T 5ee738571af740c1b494d4f17413d1d8 P 8b8b3ad3aa8f4053b2167dd550f2086f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f9cb72be98941b5a8e504cbe34132b9, 8b8b3ad3aa8f4053b2167dd550f2086f; no voters:
I20240501 07:58:56.231106 7306 raft_consensus.cc:2781] T 5ee738571af740c1b494d4f17413d1d8 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:56.231197 7306 raft_consensus.cc:686] T 5ee738571af740c1b494d4f17413d1d8 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 LEADER]: Becoming Leader. State: Replica: 8b8b3ad3aa8f4053b2167dd550f2086f, State: Running, Role: LEADER
I20240501 07:58:56.231338 7306 consensus_queue.cc:237] T 5ee738571af740c1b494d4f17413d1d8 P 8b8b3ad3aa8f4053b2167dd550f2086f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:58:56.232100 6577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ee738571af740c1b494d4f17413d1d8" candidate_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" is_pre_election: true
I20240501 07:58:56.232201 6576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ee738571af740c1b494d4f17413d1d8" candidate_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9"
I20240501 07:58:56.232239 6577 raft_consensus.cc:2454] T 5ee738571af740c1b494d4f17413d1d8 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b8b3ad3aa8f4053b2167dd550f2086f in term 0.
I20240501 07:58:56.233112 6164 catalog_manager.cc:5484] T 5ee738571af740c1b494d4f17413d1d8 P 8b8b3ad3aa8f4053b2167dd550f2086f reported cstate change: term changed from 0 to 1, leader changed from <none> to 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129). New cstate: current_term: 1 leader_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } health_report { overall_health: UNKNOWN } } }
I20240501 07:58:56.306713 7313 raft_consensus.cc:482] T 8663a511521644009fc360ad92c093a1 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:56.306875 7313 raft_consensus.cc:504] T 8663a511521644009fc360ad92c093a1 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:58:56.307298 7313 leader_election.cc:288] T 8663a511521644009fc360ad92c093a1 P 1f9cb72be98941b5a8e504cbe34132b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130:32941), 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131:34471)
I20240501 07:58:56.307737 6577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8663a511521644009fc360ad92c093a1" candidate_uuid: "1f9cb72be98941b5a8e504cbe34132b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" is_pre_election: true
I20240501 07:58:56.307749 6446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8663a511521644009fc360ad92c093a1" candidate_uuid: "1f9cb72be98941b5a8e504cbe34132b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8315d22c56f7439a9e1fc82f153f018c" is_pre_election: true
I20240501 07:58:56.307874 6577 raft_consensus.cc:2454] T 8663a511521644009fc360ad92c093a1 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f9cb72be98941b5a8e504cbe34132b9 in term 0.
I20240501 07:58:56.307893 6446 raft_consensus.cc:2454] T 8663a511521644009fc360ad92c093a1 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f9cb72be98941b5a8e504cbe34132b9 in term 0.
I20240501 07:58:56.308205 6644 leader_election.cc:302] T 8663a511521644009fc360ad92c093a1 P 1f9cb72be98941b5a8e504cbe34132b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f9cb72be98941b5a8e504cbe34132b9, 3b164cbbc00f4e8eb3254ddd349a3ff9; no voters:
I20240501 07:58:56.308387 7313 raft_consensus.cc:2781] T 8663a511521644009fc360ad92c093a1 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:56.308456 7313 raft_consensus.cc:482] T 8663a511521644009fc360ad92c093a1 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:56.308506 7313 raft_consensus.cc:3037] T 8663a511521644009fc360ad92c093a1 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:56.309335 7313 raft_consensus.cc:504] T 8663a511521644009fc360ad92c093a1 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:58:56.309659 7313 leader_election.cc:288] T 8663a511521644009fc360ad92c093a1 P 1f9cb72be98941b5a8e504cbe34132b9 [CANDIDATE]: Term 1 election: Requested vote from peers 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130:32941), 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131:34471)
I20240501 07:58:56.309981 6446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8663a511521644009fc360ad92c093a1" candidate_uuid: "1f9cb72be98941b5a8e504cbe34132b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8315d22c56f7439a9e1fc82f153f018c"
I20240501 07:58:56.310086 6446 raft_consensus.cc:3037] T 8663a511521644009fc360ad92c093a1 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:56.310065 6577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8663a511521644009fc360ad92c093a1" candidate_uuid: "1f9cb72be98941b5a8e504cbe34132b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9"
I20240501 07:58:56.310205 6577 raft_consensus.cc:3037] T 8663a511521644009fc360ad92c093a1 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:56.310839 6446 raft_consensus.cc:2454] T 8663a511521644009fc360ad92c093a1 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f9cb72be98941b5a8e504cbe34132b9 in term 1.
I20240501 07:58:56.311019 6577 raft_consensus.cc:2454] T 8663a511521644009fc360ad92c093a1 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f9cb72be98941b5a8e504cbe34132b9 in term 1.
I20240501 07:58:56.311113 6643 leader_election.cc:302] T 8663a511521644009fc360ad92c093a1 P 1f9cb72be98941b5a8e504cbe34132b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f9cb72be98941b5a8e504cbe34132b9, 8315d22c56f7439a9e1fc82f153f018c; no voters:
I20240501 07:58:56.311293 7313 raft_consensus.cc:2781] T 8663a511521644009fc360ad92c093a1 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:56.311383 7313 raft_consensus.cc:686] T 8663a511521644009fc360ad92c093a1 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 LEADER]: Becoming Leader. State: Replica: 1f9cb72be98941b5a8e504cbe34132b9, State: Running, Role: LEADER
I20240501 07:58:56.311529 7313 consensus_queue.cc:237] T 8663a511521644009fc360ad92c093a1 P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:58:56.313133 6164 catalog_manager.cc:5484] T 8663a511521644009fc360ad92c093a1 P 1f9cb72be98941b5a8e504cbe34132b9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132). New cstate: current_term: 1 leader_uuid: "1f9cb72be98941b5a8e504cbe34132b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } health_report { overall_health: HEALTHY } } }
I20240501 07:58:56.521831 7433 raft_consensus.cc:482] T 782e37a4ebb34a68b461a74115ae8703 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:56.521977 7433 raft_consensus.cc:504] T 782e37a4ebb34a68b461a74115ae8703 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } }
I20240501 07:58:56.522552 7433 leader_election.cc:288] T 782e37a4ebb34a68b461a74115ae8703 P 8315d22c56f7439a9e1fc82f153f018c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131:34471), 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129:46875)
I20240501 07:58:56.523005 6577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "782e37a4ebb34a68b461a74115ae8703" candidate_uuid: "8315d22c56f7439a9e1fc82f153f018c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" is_pre_election: true
I20240501 07:58:56.523175 6577 raft_consensus.cc:2454] T 782e37a4ebb34a68b461a74115ae8703 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8315d22c56f7439a9e1fc82f153f018c in term 0.
I20240501 07:58:56.523525 6382 leader_election.cc:302] T 782e37a4ebb34a68b461a74115ae8703 P 8315d22c56f7439a9e1fc82f153f018c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b164cbbc00f4e8eb3254ddd349a3ff9, 8315d22c56f7439a9e1fc82f153f018c; no voters:
I20240501 07:58:56.523715 7433 raft_consensus.cc:2781] T 782e37a4ebb34a68b461a74115ae8703 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:56.523780 7433 raft_consensus.cc:482] T 782e37a4ebb34a68b461a74115ae8703 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:56.523818 7433 raft_consensus.cc:3037] T 782e37a4ebb34a68b461a74115ae8703 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:56.524679 7433 raft_consensus.cc:504] T 782e37a4ebb34a68b461a74115ae8703 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } }
I20240501 07:58:56.525075 7433 leader_election.cc:288] T 782e37a4ebb34a68b461a74115ae8703 P 8315d22c56f7439a9e1fc82f153f018c [CANDIDATE]: Term 1 election: Requested vote from peers 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131:34471), 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129:46875)
I20240501 07:58:56.525381 6577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "782e37a4ebb34a68b461a74115ae8703" candidate_uuid: "8315d22c56f7439a9e1fc82f153f018c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9"
I20240501 07:58:56.525508 6577 raft_consensus.cc:3037] T 782e37a4ebb34a68b461a74115ae8703 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:56.526284 6577 raft_consensus.cc:2454] T 782e37a4ebb34a68b461a74115ae8703 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8315d22c56f7439a9e1fc82f153f018c in term 1.
I20240501 07:58:56.526517 6314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "782e37a4ebb34a68b461a74115ae8703" candidate_uuid: "8315d22c56f7439a9e1fc82f153f018c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" is_pre_election: true
I20240501 07:58:56.526556 6382 leader_election.cc:302] T 782e37a4ebb34a68b461a74115ae8703 P 8315d22c56f7439a9e1fc82f153f018c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b164cbbc00f4e8eb3254ddd349a3ff9, 8315d22c56f7439a9e1fc82f153f018c; no voters:
I20240501 07:58:56.526664 6314 raft_consensus.cc:2454] T 782e37a4ebb34a68b461a74115ae8703 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8315d22c56f7439a9e1fc82f153f018c in term 0.
I20240501 07:58:56.526723 7433 raft_consensus.cc:2781] T 782e37a4ebb34a68b461a74115ae8703 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:56.526798 7433 raft_consensus.cc:686] T 782e37a4ebb34a68b461a74115ae8703 P 8315d22c56f7439a9e1fc82f153f018c [term 1 LEADER]: Becoming Leader. State: Replica: 8315d22c56f7439a9e1fc82f153f018c, State: Running, Role: LEADER
I20240501 07:58:56.526928 6315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "782e37a4ebb34a68b461a74115ae8703" candidate_uuid: "8315d22c56f7439a9e1fc82f153f018c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f"
I20240501 07:58:56.526957 7433 consensus_queue.cc:237] T 782e37a4ebb34a68b461a74115ae8703 P 8315d22c56f7439a9e1fc82f153f018c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } }
I20240501 07:58:56.527026 6315 raft_consensus.cc:3037] T 782e37a4ebb34a68b461a74115ae8703 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:56.527957 6315 raft_consensus.cc:2454] T 782e37a4ebb34a68b461a74115ae8703 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8315d22c56f7439a9e1fc82f153f018c in term 1.
I20240501 07:58:56.528728 6164 catalog_manager.cc:5484] T 782e37a4ebb34a68b461a74115ae8703 P 8315d22c56f7439a9e1fc82f153f018c reported cstate change: term changed from 0 to 1, leader changed from <none> to 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130). New cstate: current_term: 1 leader_uuid: "8315d22c56f7439a9e1fc82f153f018c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } health_report { overall_health: UNKNOWN } } }
W20240501 07:58:56.683477 7539 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:58:56.683732 7539 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:58:56.687101 7539 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 07:58:56.692479 6577 raft_consensus.cc:3037] T 5ee738571af740c1b494d4f17413d1d8 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:56.693837 7464 consensus_queue.cc:1035] T 5ee738571af740c1b494d4f17413d1d8 P 8b8b3ad3aa8f4053b2167dd550f2086f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:56.697487 7464 consensus_queue.cc:1035] T 5ee738571af740c1b494d4f17413d1d8 P 8b8b3ad3aa8f4053b2167dd550f2086f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_3'
I20240501 07:58:56.748814 7467 consensus_queue.cc:1035] T 8663a511521644009fc360ad92c093a1 P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:56.752940 7313 consensus_queue.cc:1035] T 8663a511521644009fc360ad92c093a1 P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:56.775851 6314 raft_consensus.cc:1261] T 782e37a4ebb34a68b461a74115ae8703 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Refusing update from remote peer 8315d22c56f7439a9e1fc82f153f018c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:56.776392 7433 consensus_queue.cc:1035] T 782e37a4ebb34a68b461a74115ae8703 P 8315d22c56f7439a9e1fc82f153f018c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:56.776917 6577 raft_consensus.cc:1261] T 782e37a4ebb34a68b461a74115ae8703 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Refusing update from remote peer 8315d22c56f7439a9e1fc82f153f018c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:56.778362 7433 consensus_queue.cc:1035] T 782e37a4ebb34a68b461a74115ae8703 P 8315d22c56f7439a9e1fc82f153f018c [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 111.552 ms
time per row: 0.055776 ms
I20240501 07:58:56.843865 6173 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55950:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 07:58:56.849315 6424 tablet_service.cc:1467] Processing CreateTablet for tablet 72b5270ae4704620be0a413139f41d26 (DEFAULT_TABLE table=rebalance_test_table_4 [id=e3d01900e54c4eb1920ccd2ff7a4f5db]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:56.849475 6295 tablet_service.cc:1467] Processing CreateTablet for tablet 72b5270ae4704620be0a413139f41d26 (DEFAULT_TABLE table=rebalance_test_table_4 [id=e3d01900e54c4eb1920ccd2ff7a4f5db]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:56.849557 6426 tablet_service.cc:1467] Processing CreateTablet for tablet 5f0df3de857c49e48a52baa71163d161 (DEFAULT_TABLE table=rebalance_test_table_4 [id=e3d01900e54c4eb1920ccd2ff7a4f5db]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:56.849653 6424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72b5270ae4704620be0a413139f41d26. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:56.849769 6426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f0df3de857c49e48a52baa71163d161. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:56.849785 6295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72b5270ae4704620be0a413139f41d26. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:56.850505 6557 tablet_service.cc:1467] Processing CreateTablet for tablet 72b5270ae4704620be0a413139f41d26 (DEFAULT_TABLE table=rebalance_test_table_4 [id=e3d01900e54c4eb1920ccd2ff7a4f5db]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:56.850760 6557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72b5270ae4704620be0a413139f41d26. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:56.851716 6556 tablet_service.cc:1467] Processing CreateTablet for tablet 810a2791eb04415dae47e85db87352cb (DEFAULT_TABLE table=rebalance_test_table_4 [id=e3d01900e54c4eb1920ccd2ff7a4f5db]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:56.851948 6556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 810a2791eb04415dae47e85db87352cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:56.852082 7604 tablet_bootstrap.cc:492] T 72b5270ae4704620be0a413139f41d26 P 8b8b3ad3aa8f4053b2167dd550f2086f: Bootstrap starting.
I20240501 07:58:56.853174 6294 tablet_service.cc:1467] Processing CreateTablet for tablet 810a2791eb04415dae47e85db87352cb (DEFAULT_TABLE table=rebalance_test_table_4 [id=e3d01900e54c4eb1920ccd2ff7a4f5db]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:56.853395 6294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 810a2791eb04415dae47e85db87352cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:56.853753 6295 tablet_service.cc:1467] Processing CreateTablet for tablet 5f0df3de857c49e48a52baa71163d161 (DEFAULT_TABLE table=rebalance_test_table_4 [id=e3d01900e54c4eb1920ccd2ff7a4f5db]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:56.853941 7603 tablet_bootstrap.cc:492] T 72b5270ae4704620be0a413139f41d26 P 8315d22c56f7439a9e1fc82f153f018c: Bootstrap starting.
I20240501 07:58:56.854007 6295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f0df3de857c49e48a52baa71163d161. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:56.854668 6687 tablet_service.cc:1467] Processing CreateTablet for tablet 5f0df3de857c49e48a52baa71163d161 (DEFAULT_TABLE table=rebalance_test_table_4 [id=e3d01900e54c4eb1920ccd2ff7a4f5db]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:56.854941 6687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f0df3de857c49e48a52baa71163d161. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:56.855058 7603 tablet_bootstrap.cc:654] T 72b5270ae4704620be0a413139f41d26 P 8315d22c56f7439a9e1fc82f153f018c: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:56.855571 6688 tablet_service.cc:1467] Processing CreateTablet for tablet 810a2791eb04415dae47e85db87352cb (DEFAULT_TABLE table=rebalance_test_table_4 [id=e3d01900e54c4eb1920ccd2ff7a4f5db]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:56.855788 6688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 810a2791eb04415dae47e85db87352cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:56.856500 7604 tablet_bootstrap.cc:654] T 72b5270ae4704620be0a413139f41d26 P 8b8b3ad3aa8f4053b2167dd550f2086f: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:56.857123 7603 tablet_bootstrap.cc:492] T 72b5270ae4704620be0a413139f41d26 P 8315d22c56f7439a9e1fc82f153f018c: No bootstrap required, opened a new log
I20240501 07:58:56.857218 7603 ts_tablet_manager.cc:1367] T 72b5270ae4704620be0a413139f41d26 P 8315d22c56f7439a9e1fc82f153f018c: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 07:58:56.857376 7605 tablet_bootstrap.cc:492] T 72b5270ae4704620be0a413139f41d26 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Bootstrap starting.
I20240501 07:58:56.857721 7603 raft_consensus.cc:348] T 72b5270ae4704620be0a413139f41d26 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } }
I20240501 07:58:56.857848 7603 raft_consensus.cc:374] T 72b5270ae4704620be0a413139f41d26 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:56.857887 7603 raft_consensus.cc:729] T 72b5270ae4704620be0a413139f41d26 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8315d22c56f7439a9e1fc82f153f018c, State: Initialized, Role: FOLLOWER
I20240501 07:58:56.857816 7607 tablet_bootstrap.cc:492] T 5f0df3de857c49e48a52baa71163d161 P 1f9cb72be98941b5a8e504cbe34132b9: Bootstrap starting.
I20240501 07:58:56.858042 7603 consensus_queue.cc:260] T 72b5270ae4704620be0a413139f41d26 P 8315d22c56f7439a9e1fc82f153f018c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } }
I20240501 07:58:56.858301 7605 tablet_bootstrap.cc:654] T 72b5270ae4704620be0a413139f41d26 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:56.858361 7603 ts_tablet_manager.cc:1398] T 72b5270ae4704620be0a413139f41d26 P 8315d22c56f7439a9e1fc82f153f018c: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 07:58:56.858526 7603 tablet_bootstrap.cc:492] T 5f0df3de857c49e48a52baa71163d161 P 8315d22c56f7439a9e1fc82f153f018c: Bootstrap starting.
I20240501 07:58:56.859076 7607 tablet_bootstrap.cc:654] T 5f0df3de857c49e48a52baa71163d161 P 1f9cb72be98941b5a8e504cbe34132b9: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:56.859328 7605 tablet_bootstrap.cc:492] T 72b5270ae4704620be0a413139f41d26 P 3b164cbbc00f4e8eb3254ddd349a3ff9: No bootstrap required, opened a new log
I20240501 07:58:56.859417 7605 ts_tablet_manager.cc:1367] T 72b5270ae4704620be0a413139f41d26 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 07:58:56.859467 7603 tablet_bootstrap.cc:654] T 5f0df3de857c49e48a52baa71163d161 P 8315d22c56f7439a9e1fc82f153f018c: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:56.859894 7605 raft_consensus.cc:348] T 72b5270ae4704620be0a413139f41d26 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } }
I20240501 07:58:56.860013 7605 raft_consensus.cc:374] T 72b5270ae4704620be0a413139f41d26 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:56.860056 7605 raft_consensus.cc:729] T 72b5270ae4704620be0a413139f41d26 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b164cbbc00f4e8eb3254ddd349a3ff9, State: Initialized, Role: FOLLOWER
I20240501 07:58:56.860190 7605 consensus_queue.cc:260] T 72b5270ae4704620be0a413139f41d26 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } }
I20240501 07:58:56.860460 7605 ts_tablet_manager.cc:1398] T 72b5270ae4704620be0a413139f41d26 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:56.860643 7605 tablet_bootstrap.cc:492] T 810a2791eb04415dae47e85db87352cb P 3b164cbbc00f4e8eb3254ddd349a3ff9: Bootstrap starting.
I20240501 07:58:56.860659 7604 tablet_bootstrap.cc:492] T 72b5270ae4704620be0a413139f41d26 P 8b8b3ad3aa8f4053b2167dd550f2086f: No bootstrap required, opened a new log
I20240501 07:58:56.860817 7604 ts_tablet_manager.cc:1367] T 72b5270ae4704620be0a413139f41d26 P 8b8b3ad3aa8f4053b2167dd550f2086f: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240501 07:58:56.860888 7603 tablet_bootstrap.cc:492] T 5f0df3de857c49e48a52baa71163d161 P 8315d22c56f7439a9e1fc82f153f018c: No bootstrap required, opened a new log
I20240501 07:58:56.860988 7603 ts_tablet_manager.cc:1367] T 5f0df3de857c49e48a52baa71163d161 P 8315d22c56f7439a9e1fc82f153f018c: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240501 07:58:56.861413 7603 raft_consensus.cc:348] T 5f0df3de857c49e48a52baa71163d161 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } }
I20240501 07:58:56.861411 7604 raft_consensus.cc:348] T 72b5270ae4704620be0a413139f41d26 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } }
I20240501 07:58:56.861517 7603 raft_consensus.cc:374] T 5f0df3de857c49e48a52baa71163d161 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:56.861555 7603 raft_consensus.cc:729] T 5f0df3de857c49e48a52baa71163d161 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8315d22c56f7439a9e1fc82f153f018c, State: Initialized, Role: FOLLOWER
I20240501 07:58:56.861635 7604 raft_consensus.cc:374] T 72b5270ae4704620be0a413139f41d26 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:56.861711 7604 raft_consensus.cc:729] T 72b5270ae4704620be0a413139f41d26 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b8b3ad3aa8f4053b2167dd550f2086f, State: Initialized, Role: FOLLOWER
I20240501 07:58:56.861673 7603 consensus_queue.cc:260] T 5f0df3de857c49e48a52baa71163d161 P 8315d22c56f7439a9e1fc82f153f018c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } }
I20240501 07:58:56.861917 7604 consensus_queue.cc:260] T 72b5270ae4704620be0a413139f41d26 P 8b8b3ad3aa8f4053b2167dd550f2086f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } }
I20240501 07:58:56.861972 7603 ts_tablet_manager.cc:1398] T 5f0df3de857c49e48a52baa71163d161 P 8315d22c56f7439a9e1fc82f153f018c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:56.862026 7605 tablet_bootstrap.cc:654] T 810a2791eb04415dae47e85db87352cb P 3b164cbbc00f4e8eb3254ddd349a3ff9: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:56.862365 7604 ts_tablet_manager.cc:1398] T 72b5270ae4704620be0a413139f41d26 P 8b8b3ad3aa8f4053b2167dd550f2086f: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 07:58:56.862531 7607 tablet_bootstrap.cc:492] T 5f0df3de857c49e48a52baa71163d161 P 1f9cb72be98941b5a8e504cbe34132b9: No bootstrap required, opened a new log
I20240501 07:58:56.862576 7604 tablet_bootstrap.cc:492] T 810a2791eb04415dae47e85db87352cb P 8b8b3ad3aa8f4053b2167dd550f2086f: Bootstrap starting.
I20240501 07:58:56.862622 7607 ts_tablet_manager.cc:1367] T 5f0df3de857c49e48a52baa71163d161 P 1f9cb72be98941b5a8e504cbe34132b9: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240501 07:58:56.863049 7607 raft_consensus.cc:348] T 5f0df3de857c49e48a52baa71163d161 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } }
I20240501 07:58:56.863152 7607 raft_consensus.cc:374] T 5f0df3de857c49e48a52baa71163d161 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:56.863183 7607 raft_consensus.cc:729] T 5f0df3de857c49e48a52baa71163d161 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f9cb72be98941b5a8e504cbe34132b9, State: Initialized, Role: FOLLOWER
I20240501 07:58:56.863305 7607 consensus_queue.cc:260] T 5f0df3de857c49e48a52baa71163d161 P 1f9cb72be98941b5a8e504cbe34132b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } }
I20240501 07:58:56.863464 7604 tablet_bootstrap.cc:654] T 810a2791eb04415dae47e85db87352cb P 8b8b3ad3aa8f4053b2167dd550f2086f: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:56.863613 7607 ts_tablet_manager.cc:1398] T 5f0df3de857c49e48a52baa71163d161 P 1f9cb72be98941b5a8e504cbe34132b9: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:58:56.863749 7607 tablet_bootstrap.cc:492] T 810a2791eb04415dae47e85db87352cb P 1f9cb72be98941b5a8e504cbe34132b9: Bootstrap starting.
I20240501 07:58:56.864802 7607 tablet_bootstrap.cc:654] T 810a2791eb04415dae47e85db87352cb P 1f9cb72be98941b5a8e504cbe34132b9: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:56.865509 7604 tablet_bootstrap.cc:492] T 810a2791eb04415dae47e85db87352cb P 8b8b3ad3aa8f4053b2167dd550f2086f: No bootstrap required, opened a new log
I20240501 07:58:56.865634 7604 ts_tablet_manager.cc:1367] T 810a2791eb04415dae47e85db87352cb P 8b8b3ad3aa8f4053b2167dd550f2086f: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20240501 07:58:56.866164 7607 tablet_bootstrap.cc:492] T 810a2791eb04415dae47e85db87352cb P 1f9cb72be98941b5a8e504cbe34132b9: No bootstrap required, opened a new log
I20240501 07:58:56.866242 7607 ts_tablet_manager.cc:1367] T 810a2791eb04415dae47e85db87352cb P 1f9cb72be98941b5a8e504cbe34132b9: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20240501 07:58:56.866290 7605 tablet_bootstrap.cc:492] T 810a2791eb04415dae47e85db87352cb P 3b164cbbc00f4e8eb3254ddd349a3ff9: No bootstrap required, opened a new log
I20240501 07:58:56.866350 7605 ts_tablet_manager.cc:1367] T 810a2791eb04415dae47e85db87352cb P 3b164cbbc00f4e8eb3254ddd349a3ff9: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 07:58:56.866657 7605 raft_consensus.cc:348] T 810a2791eb04415dae47e85db87352cb P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } }
I20240501 07:58:56.866737 7605 raft_consensus.cc:374] T 810a2791eb04415dae47e85db87352cb P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:56.866676 7607 raft_consensus.cc:348] T 810a2791eb04415dae47e85db87352cb P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } }
I20240501 07:58:56.866758 7605 raft_consensus.cc:729] T 810a2791eb04415dae47e85db87352cb P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b164cbbc00f4e8eb3254ddd349a3ff9, State: Initialized, Role: FOLLOWER
I20240501 07:58:56.866768 7607 raft_consensus.cc:374] T 810a2791eb04415dae47e85db87352cb P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:56.866806 7607 raft_consensus.cc:729] T 810a2791eb04415dae47e85db87352cb P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f9cb72be98941b5a8e504cbe34132b9, State: Initialized, Role: FOLLOWER
I20240501 07:58:56.866873 7605 consensus_queue.cc:260] T 810a2791eb04415dae47e85db87352cb P 3b164cbbc00f4e8eb3254ddd349a3ff9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } }
I20240501 07:58:56.866912 7607 consensus_queue.cc:260] T 810a2791eb04415dae47e85db87352cb P 1f9cb72be98941b5a8e504cbe34132b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } }
I20240501 07:58:56.867162 7605 ts_tablet_manager.cc:1398] T 810a2791eb04415dae47e85db87352cb P 3b164cbbc00f4e8eb3254ddd349a3ff9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:56.867180 7607 ts_tablet_manager.cc:1398] T 810a2791eb04415dae47e85db87352cb P 1f9cb72be98941b5a8e504cbe34132b9: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:58:56.867882 7604 raft_consensus.cc:348] T 810a2791eb04415dae47e85db87352cb P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } }
I20240501 07:58:56.867991 7604 raft_consensus.cc:374] T 810a2791eb04415dae47e85db87352cb P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:56.868026 7604 raft_consensus.cc:729] T 810a2791eb04415dae47e85db87352cb P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b8b3ad3aa8f4053b2167dd550f2086f, State: Initialized, Role: FOLLOWER
I20240501 07:58:56.868144 7604 consensus_queue.cc:260] T 810a2791eb04415dae47e85db87352cb P 8b8b3ad3aa8f4053b2167dd550f2086f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } }
I20240501 07:58:56.868479 7604 ts_tablet_manager.cc:1398] T 810a2791eb04415dae47e85db87352cb P 8b8b3ad3aa8f4053b2167dd550f2086f: Time spent starting tablet: real 0.003s user 0.000s sys 0.000s
I20240501 07:58:56.869030 7604 tablet_bootstrap.cc:492] T 5f0df3de857c49e48a52baa71163d161 P 8b8b3ad3aa8f4053b2167dd550f2086f: Bootstrap starting.
I20240501 07:58:56.869911 7604 tablet_bootstrap.cc:654] T 5f0df3de857c49e48a52baa71163d161 P 8b8b3ad3aa8f4053b2167dd550f2086f: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:56.870978 7604 tablet_bootstrap.cc:492] T 5f0df3de857c49e48a52baa71163d161 P 8b8b3ad3aa8f4053b2167dd550f2086f: No bootstrap required, opened a new log
I20240501 07:58:56.871068 7604 ts_tablet_manager.cc:1367] T 5f0df3de857c49e48a52baa71163d161 P 8b8b3ad3aa8f4053b2167dd550f2086f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 07:58:56.871505 7604 raft_consensus.cc:348] T 5f0df3de857c49e48a52baa71163d161 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } }
I20240501 07:58:56.871593 7604 raft_consensus.cc:374] T 5f0df3de857c49e48a52baa71163d161 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:56.871634 7604 raft_consensus.cc:729] T 5f0df3de857c49e48a52baa71163d161 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b8b3ad3aa8f4053b2167dd550f2086f, State: Initialized, Role: FOLLOWER
I20240501 07:58:56.871753 7604 consensus_queue.cc:260] T 5f0df3de857c49e48a52baa71163d161 P 8b8b3ad3aa8f4053b2167dd550f2086f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } }
I20240501 07:58:56.872014 7604 ts_tablet_manager.cc:1398] T 5f0df3de857c49e48a52baa71163d161 P 8b8b3ad3aa8f4053b2167dd550f2086f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:58:56.882570 7594 raft_consensus.cc:482] T 810a2791eb04415dae47e85db87352cb P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:56.882690 7594 raft_consensus.cc:504] T 810a2791eb04415dae47e85db87352cb P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } }
I20240501 07:58:56.883023 7594 leader_election.cc:288] T 810a2791eb04415dae47e85db87352cb P 1f9cb72be98941b5a8e504cbe34132b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131:34471), 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129:46875)
I20240501 07:58:56.883342 6314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "810a2791eb04415dae47e85db87352cb" candidate_uuid: "1f9cb72be98941b5a8e504cbe34132b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" is_pre_election: true
I20240501 07:58:56.883356 6577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "810a2791eb04415dae47e85db87352cb" candidate_uuid: "1f9cb72be98941b5a8e504cbe34132b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" is_pre_election: true
I20240501 07:58:56.883498 6577 raft_consensus.cc:2454] T 810a2791eb04415dae47e85db87352cb P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f9cb72be98941b5a8e504cbe34132b9 in term 0.
I20240501 07:58:56.883497 6314 raft_consensus.cc:2454] T 810a2791eb04415dae47e85db87352cb P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f9cb72be98941b5a8e504cbe34132b9 in term 0.
I20240501 07:58:56.883780 6641 leader_election.cc:302] T 810a2791eb04415dae47e85db87352cb P 1f9cb72be98941b5a8e504cbe34132b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f9cb72be98941b5a8e504cbe34132b9, 8b8b3ad3aa8f4053b2167dd550f2086f; no voters:
I20240501 07:58:56.883975 7594 raft_consensus.cc:2781] T 810a2791eb04415dae47e85db87352cb P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:56.884028 7594 raft_consensus.cc:482] T 810a2791eb04415dae47e85db87352cb P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:56.884054 7594 raft_consensus.cc:3037] T 810a2791eb04415dae47e85db87352cb P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:56.884828 7594 raft_consensus.cc:504] T 810a2791eb04415dae47e85db87352cb P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } }
I20240501 07:58:56.885160 7594 leader_election.cc:288] T 810a2791eb04415dae47e85db87352cb P 1f9cb72be98941b5a8e504cbe34132b9 [CANDIDATE]: Term 1 election: Requested vote from peers 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131:34471), 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129:46875)
I20240501 07:58:56.885488 6314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "810a2791eb04415dae47e85db87352cb" candidate_uuid: "1f9cb72be98941b5a8e504cbe34132b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f"
I20240501 07:58:56.885519 6577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "810a2791eb04415dae47e85db87352cb" candidate_uuid: "1f9cb72be98941b5a8e504cbe34132b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9"
I20240501 07:58:56.885599 6314 raft_consensus.cc:3037] T 810a2791eb04415dae47e85db87352cb P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:56.885613 6577 raft_consensus.cc:3037] T 810a2791eb04415dae47e85db87352cb P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:56.886545 6577 raft_consensus.cc:2454] T 810a2791eb04415dae47e85db87352cb P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f9cb72be98941b5a8e504cbe34132b9 in term 1.
I20240501 07:58:56.886545 6314 raft_consensus.cc:2454] T 810a2791eb04415dae47e85db87352cb P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f9cb72be98941b5a8e504cbe34132b9 in term 1.
I20240501 07:58:56.886819 6644 leader_election.cc:302] T 810a2791eb04415dae47e85db87352cb P 1f9cb72be98941b5a8e504cbe34132b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f9cb72be98941b5a8e504cbe34132b9, 3b164cbbc00f4e8eb3254ddd349a3ff9; no voters:
I20240501 07:58:56.887002 7594 raft_consensus.cc:2781] T 810a2791eb04415dae47e85db87352cb P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:56.887094 7594 raft_consensus.cc:686] T 810a2791eb04415dae47e85db87352cb P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 LEADER]: Becoming Leader. State: Replica: 1f9cb72be98941b5a8e504cbe34132b9, State: Running, Role: LEADER
I20240501 07:58:56.887235 7594 consensus_queue.cc:237] T 810a2791eb04415dae47e85db87352cb P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } }
I20240501 07:58:56.888777 6171 catalog_manager.cc:5484] T 810a2791eb04415dae47e85db87352cb P 1f9cb72be98941b5a8e504cbe34132b9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132). New cstate: current_term: 1 leader_uuid: "1f9cb72be98941b5a8e504cbe34132b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } health_report { overall_health: UNKNOWN } } }
I20240501 07:58:56.938912 7472 raft_consensus.cc:482] T 72b5270ae4704620be0a413139f41d26 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:56.939044 7472 raft_consensus.cc:504] T 72b5270ae4704620be0a413139f41d26 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } }
I20240501 07:58:56.939448 7472 leader_election.cc:288] T 72b5270ae4704620be0a413139f41d26 P 8315d22c56f7439a9e1fc82f153f018c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129:46875), 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131:34471)
I20240501 07:58:56.939850 6577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72b5270ae4704620be0a413139f41d26" candidate_uuid: "8315d22c56f7439a9e1fc82f153f018c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" is_pre_election: true
I20240501 07:58:56.939885 6314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72b5270ae4704620be0a413139f41d26" candidate_uuid: "8315d22c56f7439a9e1fc82f153f018c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" is_pre_election: true
I20240501 07:58:56.939996 6577 raft_consensus.cc:2454] T 72b5270ae4704620be0a413139f41d26 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8315d22c56f7439a9e1fc82f153f018c in term 0.
I20240501 07:58:56.940057 6314 raft_consensus.cc:2454] T 72b5270ae4704620be0a413139f41d26 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8315d22c56f7439a9e1fc82f153f018c in term 0.
I20240501 07:58:56.940337 6382 leader_election.cc:302] T 72b5270ae4704620be0a413139f41d26 P 8315d22c56f7439a9e1fc82f153f018c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b164cbbc00f4e8eb3254ddd349a3ff9, 8315d22c56f7439a9e1fc82f153f018c; no voters:
I20240501 07:58:56.940552 7472 raft_consensus.cc:2781] T 72b5270ae4704620be0a413139f41d26 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:56.940627 7472 raft_consensus.cc:482] T 72b5270ae4704620be0a413139f41d26 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:56.940655 7472 raft_consensus.cc:3037] T 72b5270ae4704620be0a413139f41d26 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:56.957805 7472 raft_consensus.cc:504] T 72b5270ae4704620be0a413139f41d26 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } }
I20240501 07:58:56.958423 7472 leader_election.cc:288] T 72b5270ae4704620be0a413139f41d26 P 8315d22c56f7439a9e1fc82f153f018c [CANDIDATE]: Term 1 election: Requested vote from peers 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129:46875), 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131:34471)
I20240501 07:58:56.958907 6577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72b5270ae4704620be0a413139f41d26" candidate_uuid: "8315d22c56f7439a9e1fc82f153f018c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9"
I20240501 07:58:56.958904 6314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72b5270ae4704620be0a413139f41d26" candidate_uuid: "8315d22c56f7439a9e1fc82f153f018c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f"
I20240501 07:58:56.959049 6577 raft_consensus.cc:3037] T 72b5270ae4704620be0a413139f41d26 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:56.959046 6314 raft_consensus.cc:3037] T 72b5270ae4704620be0a413139f41d26 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:56.959853 6577 raft_consensus.cc:2454] T 72b5270ae4704620be0a413139f41d26 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8315d22c56f7439a9e1fc82f153f018c in term 1.
I20240501 07:58:56.960176 6382 leader_election.cc:302] T 72b5270ae4704620be0a413139f41d26 P 8315d22c56f7439a9e1fc82f153f018c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b164cbbc00f4e8eb3254ddd349a3ff9, 8315d22c56f7439a9e1fc82f153f018c; no voters:
I20240501 07:58:56.960256 6314 raft_consensus.cc:2454] T 72b5270ae4704620be0a413139f41d26 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8315d22c56f7439a9e1fc82f153f018c in term 1.
I20240501 07:58:56.960366 7472 raft_consensus.cc:2781] T 72b5270ae4704620be0a413139f41d26 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:56.960435 7472 raft_consensus.cc:686] T 72b5270ae4704620be0a413139f41d26 P 8315d22c56f7439a9e1fc82f153f018c [term 1 LEADER]: Becoming Leader. State: Replica: 8315d22c56f7439a9e1fc82f153f018c, State: Running, Role: LEADER
I20240501 07:58:56.960582 7472 consensus_queue.cc:237] T 72b5270ae4704620be0a413139f41d26 P 8315d22c56f7439a9e1fc82f153f018c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } }
I20240501 07:58:56.962323 6171 catalog_manager.cc:5484] T 72b5270ae4704620be0a413139f41d26 P 8315d22c56f7439a9e1fc82f153f018c reported cstate change: term changed from 0 to 1, leader changed from <none> to 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130). New cstate: current_term: 1 leader_uuid: "8315d22c56f7439a9e1fc82f153f018c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } health_report { overall_health: UNKNOWN } } }
I20240501 07:58:57.137223 7583 raft_consensus.cc:482] T 5f0df3de857c49e48a52baa71163d161 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:57.137404 7583 raft_consensus.cc:504] T 5f0df3de857c49e48a52baa71163d161 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } }
I20240501 07:58:57.137912 7583 leader_election.cc:288] T 5f0df3de857c49e48a52baa71163d161 P 8b8b3ad3aa8f4053b2167dd550f2086f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130:32941), 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132:36773)
I20240501 07:58:57.138494 6446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f0df3de857c49e48a52baa71163d161" candidate_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8315d22c56f7439a9e1fc82f153f018c" is_pre_election: true
I20240501 07:58:57.138486 6708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f0df3de857c49e48a52baa71163d161" candidate_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f9cb72be98941b5a8e504cbe34132b9" is_pre_election: true
I20240501 07:58:57.138629 6708 raft_consensus.cc:2454] T 5f0df3de857c49e48a52baa71163d161 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b8b3ad3aa8f4053b2167dd550f2086f in term 0.
I20240501 07:58:57.138640 6446 raft_consensus.cc:2454] T 5f0df3de857c49e48a52baa71163d161 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b8b3ad3aa8f4053b2167dd550f2086f in term 0.
I20240501 07:58:57.138986 6250 leader_election.cc:302] T 5f0df3de857c49e48a52baa71163d161 P 8b8b3ad3aa8f4053b2167dd550f2086f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f9cb72be98941b5a8e504cbe34132b9, 8b8b3ad3aa8f4053b2167dd550f2086f; no voters:
I20240501 07:58:57.139171 7583 raft_consensus.cc:2781] T 5f0df3de857c49e48a52baa71163d161 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:57.139232 7583 raft_consensus.cc:482] T 5f0df3de857c49e48a52baa71163d161 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:57.139286 7583 raft_consensus.cc:3037] T 5f0df3de857c49e48a52baa71163d161 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:57.199079 7583 raft_consensus.cc:504] T 5f0df3de857c49e48a52baa71163d161 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } }
I20240501 07:58:57.199743 7583 leader_election.cc:288] T 5f0df3de857c49e48a52baa71163d161 P 8b8b3ad3aa8f4053b2167dd550f2086f [CANDIDATE]: Term 1 election: Requested vote from peers 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130:32941), 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132:36773)
I20240501 07:58:57.200279 6446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f0df3de857c49e48a52baa71163d161" candidate_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8315d22c56f7439a9e1fc82f153f018c"
I20240501 07:58:57.200279 6708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f0df3de857c49e48a52baa71163d161" candidate_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f9cb72be98941b5a8e504cbe34132b9"
I20240501 07:58:57.200453 6446 raft_consensus.cc:3037] T 5f0df3de857c49e48a52baa71163d161 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:57.200453 6708 raft_consensus.cc:3037] T 5f0df3de857c49e48a52baa71163d161 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:57.201504 6708 raft_consensus.cc:2454] T 5f0df3de857c49e48a52baa71163d161 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b8b3ad3aa8f4053b2167dd550f2086f in term 1.
I20240501 07:58:57.201503 6446 raft_consensus.cc:2454] T 5f0df3de857c49e48a52baa71163d161 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b8b3ad3aa8f4053b2167dd550f2086f in term 1.
I20240501 07:58:57.201881 6250 leader_election.cc:302] T 5f0df3de857c49e48a52baa71163d161 P 8b8b3ad3aa8f4053b2167dd550f2086f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8315d22c56f7439a9e1fc82f153f018c, 8b8b3ad3aa8f4053b2167dd550f2086f; no voters:
I20240501 07:58:57.202061 7583 raft_consensus.cc:2781] T 5f0df3de857c49e48a52baa71163d161 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:57.202443 7583 raft_consensus.cc:686] T 5f0df3de857c49e48a52baa71163d161 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 LEADER]: Becoming Leader. State: Replica: 8b8b3ad3aa8f4053b2167dd550f2086f, State: Running, Role: LEADER
I20240501 07:58:57.202715 7583 consensus_queue.cc:237] T 5f0df3de857c49e48a52baa71163d161 P 8b8b3ad3aa8f4053b2167dd550f2086f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } }
I20240501 07:58:57.205046 6171 catalog_manager.cc:5484] T 5f0df3de857c49e48a52baa71163d161 P 8b8b3ad3aa8f4053b2167dd550f2086f reported cstate change: term changed from 0 to 1, leader changed from <none> to 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129). New cstate: current_term: 1 leader_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } health_report { overall_health: HEALTHY } } }
I20240501 07:58:57.335224 7594 consensus_queue.cc:1035] T 810a2791eb04415dae47e85db87352cb P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 07:58:57.335950 7620 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:58:57.336297 7620 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240501 07:58:57.338806 7467 consensus_queue.cc:1035] T 810a2791eb04415dae47e85db87352cb P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 07:58:57.341450 7620 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
I20240501 07:58:57.377091 7472 consensus_queue.cc:1035] T 72b5270ae4704620be0a413139f41d26 P 8315d22c56f7439a9e1fc82f153f018c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:57.380563 7433 consensus_queue.cc:1035] T 72b5270ae4704620be0a413139f41d26 P 8315d22c56f7439a9e1fc82f153f018c [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:57.439579 6708 raft_consensus.cc:1261] T 5f0df3de857c49e48a52baa71163d161 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Refusing update from remote peer 8b8b3ad3aa8f4053b2167dd550f2086f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:57.439993 7583 consensus_queue.cc:1035] T 5f0df3de857c49e48a52baa71163d161 P 8b8b3ad3aa8f4053b2167dd550f2086f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:57.443534 6446 raft_consensus.cc:1261] T 5f0df3de857c49e48a52baa71163d161 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Refusing update from remote peer 8b8b3ad3aa8f4053b2167dd550f2086f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:57.443979 7583 consensus_queue.cc:1035] T 5f0df3de857c49e48a52baa71163d161 P 8b8b3ad3aa8f4053b2167dd550f2086f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 122.66 ms
time per row: 0.0613302 ms
I20240501 07:58:57.509069 6173 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55950:
name: "rebalance_test_table_5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 07:58:57.514743 6556 tablet_service.cc:1467] Processing CreateTablet for tablet 114b6dbafa1c4c44ac42f1d2293e3125 (DEFAULT_TABLE table=rebalance_test_table_5 [id=336b9f044a794b6a8a51d76e720b0ecd]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:57.515134 6556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 114b6dbafa1c4c44ac42f1d2293e3125. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:57.515456 6424 tablet_service.cc:1467] Processing CreateTablet for tablet df00e37199e944589a0a5f3feeee308a (DEFAULT_TABLE table=rebalance_test_table_5 [id=336b9f044a794b6a8a51d76e720b0ecd]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:57.515734 6424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df00e37199e944589a0a5f3feeee308a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:57.517102 7680 tablet_bootstrap.cc:492] T 114b6dbafa1c4c44ac42f1d2293e3125 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Bootstrap starting.
I20240501 07:58:57.517261 6426 tablet_service.cc:1467] Processing CreateTablet for tablet 114b6dbafa1c4c44ac42f1d2293e3125 (DEFAULT_TABLE table=rebalance_test_table_5 [id=336b9f044a794b6a8a51d76e720b0ecd]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:57.517509 6426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 114b6dbafa1c4c44ac42f1d2293e3125. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:57.518152 7680 tablet_bootstrap.cc:654] T 114b6dbafa1c4c44ac42f1d2293e3125 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:57.518117 6688 tablet_service.cc:1467] Processing CreateTablet for tablet 114b6dbafa1c4c44ac42f1d2293e3125 (DEFAULT_TABLE table=rebalance_test_table_5 [id=336b9f044a794b6a8a51d76e720b0ecd]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:57.518117 6687 tablet_service.cc:1467] Processing CreateTablet for tablet df00e37199e944589a0a5f3feeee308a (DEFAULT_TABLE table=rebalance_test_table_5 [id=336b9f044a794b6a8a51d76e720b0ecd]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:57.518441 6688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 114b6dbafa1c4c44ac42f1d2293e3125. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:57.518569 6687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df00e37199e944589a0a5f3feeee308a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:57.519567 6295 tablet_service.cc:1467] Processing CreateTablet for tablet f68dbc1f65eb4f1b91a62e2de82a7676 (DEFAULT_TABLE table=rebalance_test_table_5 [id=336b9f044a794b6a8a51d76e720b0ecd]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:57.519850 6295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f68dbc1f65eb4f1b91a62e2de82a7676. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:57.520999 7683 tablet_bootstrap.cc:492] T 114b6dbafa1c4c44ac42f1d2293e3125 P 1f9cb72be98941b5a8e504cbe34132b9: Bootstrap starting.
I20240501 07:58:57.522207 7683 tablet_bootstrap.cc:654] T 114b6dbafa1c4c44ac42f1d2293e3125 P 1f9cb72be98941b5a8e504cbe34132b9: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:57.522405 7681 tablet_bootstrap.cc:492] T df00e37199e944589a0a5f3feeee308a P 8315d22c56f7439a9e1fc82f153f018c: Bootstrap starting.
I20240501 07:58:57.522459 6426 tablet_service.cc:1467] Processing CreateTablet for tablet f68dbc1f65eb4f1b91a62e2de82a7676 (DEFAULT_TABLE table=rebalance_test_table_5 [id=336b9f044a794b6a8a51d76e720b0ecd]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:57.522545 6557 tablet_service.cc:1467] Processing CreateTablet for tablet f68dbc1f65eb4f1b91a62e2de82a7676 (DEFAULT_TABLE table=rebalance_test_table_5 [id=336b9f044a794b6a8a51d76e720b0ecd]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:57.522750 6426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f68dbc1f65eb4f1b91a62e2de82a7676. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:57.522761 6557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f68dbc1f65eb4f1b91a62e2de82a7676. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:57.523458 7681 tablet_bootstrap.cc:654] T df00e37199e944589a0a5f3feeee308a P 8315d22c56f7439a9e1fc82f153f018c: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:57.523579 7683 tablet_bootstrap.cc:492] T 114b6dbafa1c4c44ac42f1d2293e3125 P 1f9cb72be98941b5a8e504cbe34132b9: No bootstrap required, opened a new log
I20240501 07:58:57.523661 7683 ts_tablet_manager.cc:1367] T 114b6dbafa1c4c44ac42f1d2293e3125 P 1f9cb72be98941b5a8e504cbe34132b9: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 07:58:57.524107 7683 raft_consensus.cc:348] T 114b6dbafa1c4c44ac42f1d2293e3125 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:58:57.524190 7683 raft_consensus.cc:374] T 114b6dbafa1c4c44ac42f1d2293e3125 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:57.524210 7683 raft_consensus.cc:729] T 114b6dbafa1c4c44ac42f1d2293e3125 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f9cb72be98941b5a8e504cbe34132b9, State: Initialized, Role: FOLLOWER
I20240501 07:58:57.524325 7683 consensus_queue.cc:260] T 114b6dbafa1c4c44ac42f1d2293e3125 P 1f9cb72be98941b5a8e504cbe34132b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:58:57.524586 7683 ts_tablet_manager.cc:1398] T 114b6dbafa1c4c44ac42f1d2293e3125 P 1f9cb72be98941b5a8e504cbe34132b9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:57.524721 7683 tablet_bootstrap.cc:492] T df00e37199e944589a0a5f3feeee308a P 1f9cb72be98941b5a8e504cbe34132b9: Bootstrap starting.
I20240501 07:58:57.525168 6556 tablet_service.cc:1467] Processing CreateTablet for tablet df00e37199e944589a0a5f3feeee308a (DEFAULT_TABLE table=rebalance_test_table_5 [id=336b9f044a794b6a8a51d76e720b0ecd]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:57.525408 6556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df00e37199e944589a0a5f3feeee308a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:57.525612 7683 tablet_bootstrap.cc:654] T df00e37199e944589a0a5f3feeee308a P 1f9cb72be98941b5a8e504cbe34132b9: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:57.525916 7684 tablet_bootstrap.cc:492] T f68dbc1f65eb4f1b91a62e2de82a7676 P 8b8b3ad3aa8f4053b2167dd550f2086f: Bootstrap starting.
I20240501 07:58:57.526819 7683 tablet_bootstrap.cc:492] T df00e37199e944589a0a5f3feeee308a P 1f9cb72be98941b5a8e504cbe34132b9: No bootstrap required, opened a new log
I20240501 07:58:57.526883 7684 tablet_bootstrap.cc:654] T f68dbc1f65eb4f1b91a62e2de82a7676 P 8b8b3ad3aa8f4053b2167dd550f2086f: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:57.526902 7683 ts_tablet_manager.cc:1367] T df00e37199e944589a0a5f3feeee308a P 1f9cb72be98941b5a8e504cbe34132b9: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 07:58:57.527395 7683 raft_consensus.cc:348] T df00e37199e944589a0a5f3feeee308a P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:58:57.527505 7683 raft_consensus.cc:374] T df00e37199e944589a0a5f3feeee308a P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:57.527539 7683 raft_consensus.cc:729] T df00e37199e944589a0a5f3feeee308a P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f9cb72be98941b5a8e504cbe34132b9, State: Initialized, Role: FOLLOWER
I20240501 07:58:57.527648 7681 tablet_bootstrap.cc:492] T df00e37199e944589a0a5f3feeee308a P 8315d22c56f7439a9e1fc82f153f018c: No bootstrap required, opened a new log
I20240501 07:58:57.527655 7683 consensus_queue.cc:260] T df00e37199e944589a0a5f3feeee308a P 1f9cb72be98941b5a8e504cbe34132b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:58:57.527726 7681 ts_tablet_manager.cc:1367] T df00e37199e944589a0a5f3feeee308a P 8315d22c56f7439a9e1fc82f153f018c: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 07:58:57.527925 7683 ts_tablet_manager.cc:1398] T df00e37199e944589a0a5f3feeee308a P 1f9cb72be98941b5a8e504cbe34132b9: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:58:57.528162 7681 raft_consensus.cc:348] T df00e37199e944589a0a5f3feeee308a P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:58:57.528236 7681 raft_consensus.cc:374] T df00e37199e944589a0a5f3feeee308a P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:57.528256 7681 raft_consensus.cc:729] T df00e37199e944589a0a5f3feeee308a P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8315d22c56f7439a9e1fc82f153f018c, State: Initialized, Role: FOLLOWER
I20240501 07:58:57.528357 7681 consensus_queue.cc:260] T df00e37199e944589a0a5f3feeee308a P 8315d22c56f7439a9e1fc82f153f018c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:58:57.528647 7681 ts_tablet_manager.cc:1398] T df00e37199e944589a0a5f3feeee308a P 8315d22c56f7439a9e1fc82f153f018c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:57.528704 7680 tablet_bootstrap.cc:492] T 114b6dbafa1c4c44ac42f1d2293e3125 P 3b164cbbc00f4e8eb3254ddd349a3ff9: No bootstrap required, opened a new log
I20240501 07:58:57.528793 7680 ts_tablet_manager.cc:1367] T 114b6dbafa1c4c44ac42f1d2293e3125 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240501 07:58:57.528808 7681 tablet_bootstrap.cc:492] T 114b6dbafa1c4c44ac42f1d2293e3125 P 8315d22c56f7439a9e1fc82f153f018c: Bootstrap starting.
I20240501 07:58:57.529443 7680 raft_consensus.cc:348] T 114b6dbafa1c4c44ac42f1d2293e3125 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:58:57.529539 7681 tablet_bootstrap.cc:654] T 114b6dbafa1c4c44ac42f1d2293e3125 P 8315d22c56f7439a9e1fc82f153f018c: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:57.529551 7680 raft_consensus.cc:374] T 114b6dbafa1c4c44ac42f1d2293e3125 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:57.529608 7680 raft_consensus.cc:729] T 114b6dbafa1c4c44ac42f1d2293e3125 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b164cbbc00f4e8eb3254ddd349a3ff9, State: Initialized, Role: FOLLOWER
I20240501 07:58:57.529742 7680 consensus_queue.cc:260] T 114b6dbafa1c4c44ac42f1d2293e3125 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:58:57.530150 7680 ts_tablet_manager.cc:1398] T 114b6dbafa1c4c44ac42f1d2293e3125 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240501 07:58:57.530326 7680 tablet_bootstrap.cc:492] T f68dbc1f65eb4f1b91a62e2de82a7676 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Bootstrap starting.
I20240501 07:58:57.531209 7680 tablet_bootstrap.cc:654] T f68dbc1f65eb4f1b91a62e2de82a7676 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:57.531502 7681 tablet_bootstrap.cc:492] T 114b6dbafa1c4c44ac42f1d2293e3125 P 8315d22c56f7439a9e1fc82f153f018c: No bootstrap required, opened a new log
I20240501 07:58:57.531591 7681 ts_tablet_manager.cc:1367] T 114b6dbafa1c4c44ac42f1d2293e3125 P 8315d22c56f7439a9e1fc82f153f018c: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240501 07:58:57.532035 7681 raft_consensus.cc:348] T 114b6dbafa1c4c44ac42f1d2293e3125 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:58:57.532138 7681 raft_consensus.cc:374] T 114b6dbafa1c4c44ac42f1d2293e3125 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:57.532168 7681 raft_consensus.cc:729] T 114b6dbafa1c4c44ac42f1d2293e3125 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8315d22c56f7439a9e1fc82f153f018c, State: Initialized, Role: FOLLOWER
I20240501 07:58:57.532284 7684 tablet_bootstrap.cc:492] T f68dbc1f65eb4f1b91a62e2de82a7676 P 8b8b3ad3aa8f4053b2167dd550f2086f: No bootstrap required, opened a new log
I20240501 07:58:57.532271 7681 consensus_queue.cc:260] T 114b6dbafa1c4c44ac42f1d2293e3125 P 8315d22c56f7439a9e1fc82f153f018c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:58:57.532580 7681 ts_tablet_manager.cc:1398] T 114b6dbafa1c4c44ac42f1d2293e3125 P 8315d22c56f7439a9e1fc82f153f018c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:58:57.532841 7681 tablet_bootstrap.cc:492] T f68dbc1f65eb4f1b91a62e2de82a7676 P 8315d22c56f7439a9e1fc82f153f018c: Bootstrap starting.
I20240501 07:58:57.532858 7684 ts_tablet_manager.cc:1367] T f68dbc1f65eb4f1b91a62e2de82a7676 P 8b8b3ad3aa8f4053b2167dd550f2086f: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240501 07:58:57.533418 7684 raft_consensus.cc:348] T f68dbc1f65eb4f1b91a62e2de82a7676 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } }
I20240501 07:58:57.533541 7684 raft_consensus.cc:374] T f68dbc1f65eb4f1b91a62e2de82a7676 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:57.533572 7684 raft_consensus.cc:729] T f68dbc1f65eb4f1b91a62e2de82a7676 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b8b3ad3aa8f4053b2167dd550f2086f, State: Initialized, Role: FOLLOWER
I20240501 07:58:57.533738 7684 consensus_queue.cc:260] T f68dbc1f65eb4f1b91a62e2de82a7676 P 8b8b3ad3aa8f4053b2167dd550f2086f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } }
I20240501 07:58:57.533825 7681 tablet_bootstrap.cc:654] T f68dbc1f65eb4f1b91a62e2de82a7676 P 8315d22c56f7439a9e1fc82f153f018c: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:57.534019 7684 ts_tablet_manager.cc:1398] T f68dbc1f65eb4f1b91a62e2de82a7676 P 8b8b3ad3aa8f4053b2167dd550f2086f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:57.535562 7681 tablet_bootstrap.cc:492] T f68dbc1f65eb4f1b91a62e2de82a7676 P 8315d22c56f7439a9e1fc82f153f018c: No bootstrap required, opened a new log
I20240501 07:58:57.535668 7681 ts_tablet_manager.cc:1367] T f68dbc1f65eb4f1b91a62e2de82a7676 P 8315d22c56f7439a9e1fc82f153f018c: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240501 07:58:57.536386 7681 raft_consensus.cc:348] T f68dbc1f65eb4f1b91a62e2de82a7676 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } }
I20240501 07:58:57.536490 7681 raft_consensus.cc:374] T f68dbc1f65eb4f1b91a62e2de82a7676 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:57.536525 7681 raft_consensus.cc:729] T f68dbc1f65eb4f1b91a62e2de82a7676 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8315d22c56f7439a9e1fc82f153f018c, State: Initialized, Role: FOLLOWER
I20240501 07:58:57.536687 7681 consensus_queue.cc:260] T f68dbc1f65eb4f1b91a62e2de82a7676 P 8315d22c56f7439a9e1fc82f153f018c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } }
I20240501 07:58:57.536993 7681 ts_tablet_manager.cc:1398] T f68dbc1f65eb4f1b91a62e2de82a7676 P 8315d22c56f7439a9e1fc82f153f018c: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240501 07:58:57.538249 7680 tablet_bootstrap.cc:492] T f68dbc1f65eb4f1b91a62e2de82a7676 P 3b164cbbc00f4e8eb3254ddd349a3ff9: No bootstrap required, opened a new log
I20240501 07:58:57.538359 7680 ts_tablet_manager.cc:1367] T f68dbc1f65eb4f1b91a62e2de82a7676 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240501 07:58:57.538750 7680 raft_consensus.cc:348] T f68dbc1f65eb4f1b91a62e2de82a7676 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } }
I20240501 07:58:57.538836 7680 raft_consensus.cc:374] T f68dbc1f65eb4f1b91a62e2de82a7676 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:57.538863 7680 raft_consensus.cc:729] T f68dbc1f65eb4f1b91a62e2de82a7676 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b164cbbc00f4e8eb3254ddd349a3ff9, State: Initialized, Role: FOLLOWER
I20240501 07:58:57.538950 7680 consensus_queue.cc:260] T f68dbc1f65eb4f1b91a62e2de82a7676 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } }
I20240501 07:58:57.539199 7680 ts_tablet_manager.cc:1398] T f68dbc1f65eb4f1b91a62e2de82a7676 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:58:57.539376 7680 tablet_bootstrap.cc:492] T df00e37199e944589a0a5f3feeee308a P 3b164cbbc00f4e8eb3254ddd349a3ff9: Bootstrap starting.
I20240501 07:58:57.540133 7680 tablet_bootstrap.cc:654] T df00e37199e944589a0a5f3feeee308a P 3b164cbbc00f4e8eb3254ddd349a3ff9: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:57.541306 7680 tablet_bootstrap.cc:492] T df00e37199e944589a0a5f3feeee308a P 3b164cbbc00f4e8eb3254ddd349a3ff9: No bootstrap required, opened a new log
I20240501 07:58:57.541391 7680 ts_tablet_manager.cc:1367] T df00e37199e944589a0a5f3feeee308a P 3b164cbbc00f4e8eb3254ddd349a3ff9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:58:57.541745 7680 raft_consensus.cc:348] T df00e37199e944589a0a5f3feeee308a P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:58:57.541852 7680 raft_consensus.cc:374] T df00e37199e944589a0a5f3feeee308a P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:57.541898 7680 raft_consensus.cc:729] T df00e37199e944589a0a5f3feeee308a P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b164cbbc00f4e8eb3254ddd349a3ff9, State: Initialized, Role: FOLLOWER
I20240501 07:58:57.541999 7680 consensus_queue.cc:260] T df00e37199e944589a0a5f3feeee308a P 3b164cbbc00f4e8eb3254ddd349a3ff9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:58:57.542237 7680 ts_tablet_manager.cc:1398] T df00e37199e944589a0a5f3feeee308a P 3b164cbbc00f4e8eb3254ddd349a3ff9: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:58:57.690312 7472 raft_consensus.cc:482] T df00e37199e944589a0a5f3feeee308a P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:57.690452 7472 raft_consensus.cc:504] T df00e37199e944589a0a5f3feeee308a P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:58:57.690899 7472 leader_election.cc:288] T df00e37199e944589a0a5f3feeee308a P 8315d22c56f7439a9e1fc82f153f018c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131:34471), 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132:36773)
I20240501 07:58:57.691409 6708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df00e37199e944589a0a5f3feeee308a" candidate_uuid: "8315d22c56f7439a9e1fc82f153f018c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f9cb72be98941b5a8e504cbe34132b9" is_pre_election: true
I20240501 07:58:57.691417 6577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df00e37199e944589a0a5f3feeee308a" candidate_uuid: "8315d22c56f7439a9e1fc82f153f018c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" is_pre_election: true
I20240501 07:58:57.691565 6708 raft_consensus.cc:2454] T df00e37199e944589a0a5f3feeee308a P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8315d22c56f7439a9e1fc82f153f018c in term 0.
I20240501 07:58:57.691565 6577 raft_consensus.cc:2454] T df00e37199e944589a0a5f3feeee308a P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8315d22c56f7439a9e1fc82f153f018c in term 0.
I20240501 07:58:57.691893 6382 leader_election.cc:302] T df00e37199e944589a0a5f3feeee308a P 8315d22c56f7439a9e1fc82f153f018c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b164cbbc00f4e8eb3254ddd349a3ff9, 8315d22c56f7439a9e1fc82f153f018c; no voters:
I20240501 07:58:57.692075 7472 raft_consensus.cc:2781] T df00e37199e944589a0a5f3feeee308a P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:57.692128 7472 raft_consensus.cc:482] T df00e37199e944589a0a5f3feeee308a P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:57.692157 7472 raft_consensus.cc:3037] T df00e37199e944589a0a5f3feeee308a P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:57.713635 7618 raft_consensus.cc:482] T 114b6dbafa1c4c44ac42f1d2293e3125 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:57.713776 7618 raft_consensus.cc:504] T 114b6dbafa1c4c44ac42f1d2293e3125 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:58:57.714262 7618 leader_election.cc:288] T 114b6dbafa1c4c44ac42f1d2293e3125 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130:32941), 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132:36773)
I20240501 07:58:57.714783 6708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "114b6dbafa1c4c44ac42f1d2293e3125" candidate_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f9cb72be98941b5a8e504cbe34132b9" is_pre_election: true
I20240501 07:58:57.714772 6446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "114b6dbafa1c4c44ac42f1d2293e3125" candidate_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8315d22c56f7439a9e1fc82f153f018c" is_pre_election: true
I20240501 07:58:57.714931 6446 raft_consensus.cc:2454] T 114b6dbafa1c4c44ac42f1d2293e3125 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b164cbbc00f4e8eb3254ddd349a3ff9 in term 0.
I20240501 07:58:57.714931 6708 raft_consensus.cc:2454] T 114b6dbafa1c4c44ac42f1d2293e3125 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b164cbbc00f4e8eb3254ddd349a3ff9 in term 0.
I20240501 07:58:57.715230 6512 leader_election.cc:302] T 114b6dbafa1c4c44ac42f1d2293e3125 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b164cbbc00f4e8eb3254ddd349a3ff9, 8315d22c56f7439a9e1fc82f153f018c; no voters:
I20240501 07:58:57.715420 7618 raft_consensus.cc:2781] T 114b6dbafa1c4c44ac42f1d2293e3125 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:57.715479 7618 raft_consensus.cc:482] T 114b6dbafa1c4c44ac42f1d2293e3125 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:57.715531 7618 raft_consensus.cc:3037] T 114b6dbafa1c4c44ac42f1d2293e3125 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:57.749958 7472 raft_consensus.cc:504] T df00e37199e944589a0a5f3feeee308a P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:58:57.749970 7618 raft_consensus.cc:504] T 114b6dbafa1c4c44ac42f1d2293e3125 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:58:57.750474 7472 leader_election.cc:288] T df00e37199e944589a0a5f3feeee308a P 8315d22c56f7439a9e1fc82f153f018c [CANDIDATE]: Term 1 election: Requested vote from peers 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131:34471), 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132:36773)
I20240501 07:58:57.750492 7618 leader_election.cc:288] T 114b6dbafa1c4c44ac42f1d2293e3125 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [CANDIDATE]: Term 1 election: Requested vote from peers 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130:32941), 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132:36773)
I20240501 07:58:57.751232 6577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df00e37199e944589a0a5f3feeee308a" candidate_uuid: "8315d22c56f7439a9e1fc82f153f018c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9"
I20240501 07:58:57.751225 6708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df00e37199e944589a0a5f3feeee308a" candidate_uuid: "8315d22c56f7439a9e1fc82f153f018c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f9cb72be98941b5a8e504cbe34132b9"
I20240501 07:58:57.751358 6577 raft_consensus.cc:3037] T df00e37199e944589a0a5f3feeee308a P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:57.751401 6708 raft_consensus.cc:3037] T df00e37199e944589a0a5f3feeee308a P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:57.751876 6707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "114b6dbafa1c4c44ac42f1d2293e3125" candidate_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f9cb72be98941b5a8e504cbe34132b9"
I20240501 07:58:57.751986 6707 raft_consensus.cc:3037] T 114b6dbafa1c4c44ac42f1d2293e3125 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:57.752449 6577 raft_consensus.cc:2454] T df00e37199e944589a0a5f3feeee308a P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8315d22c56f7439a9e1fc82f153f018c in term 1.
I20240501 07:58:57.752586 6708 raft_consensus.cc:2454] T df00e37199e944589a0a5f3feeee308a P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8315d22c56f7439a9e1fc82f153f018c in term 1.
I20240501 07:58:57.752856 6382 leader_election.cc:302] T df00e37199e944589a0a5f3feeee308a P 8315d22c56f7439a9e1fc82f153f018c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b164cbbc00f4e8eb3254ddd349a3ff9, 8315d22c56f7439a9e1fc82f153f018c; no voters:
I20240501 07:58:57.752900 6707 raft_consensus.cc:2454] T 114b6dbafa1c4c44ac42f1d2293e3125 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b164cbbc00f4e8eb3254ddd349a3ff9 in term 1.
I20240501 07:58:57.753121 6446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "114b6dbafa1c4c44ac42f1d2293e3125" candidate_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8315d22c56f7439a9e1fc82f153f018c"
I20240501 07:58:57.753218 6512 leader_election.cc:302] T 114b6dbafa1c4c44ac42f1d2293e3125 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f9cb72be98941b5a8e504cbe34132b9, 3b164cbbc00f4e8eb3254ddd349a3ff9; no voters:
I20240501 07:58:57.753258 6446 raft_consensus.cc:3037] T 114b6dbafa1c4c44ac42f1d2293e3125 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:57.753134 7472 raft_consensus.cc:2781] T df00e37199e944589a0a5f3feeee308a P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:57.753418 7618 raft_consensus.cc:2781] T 114b6dbafa1c4c44ac42f1d2293e3125 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:57.753458 7472 raft_consensus.cc:686] T df00e37199e944589a0a5f3feeee308a P 8315d22c56f7439a9e1fc82f153f018c [term 1 LEADER]: Becoming Leader. State: Replica: 8315d22c56f7439a9e1fc82f153f018c, State: Running, Role: LEADER
I20240501 07:58:57.753612 7472 consensus_queue.cc:237] T df00e37199e944589a0a5f3feeee308a P 8315d22c56f7439a9e1fc82f153f018c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:58:57.753767 7618 raft_consensus.cc:686] T 114b6dbafa1c4c44ac42f1d2293e3125 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 LEADER]: Becoming Leader. State: Replica: 3b164cbbc00f4e8eb3254ddd349a3ff9, State: Running, Role: LEADER
I20240501 07:58:57.753895 7618 consensus_queue.cc:237] T 114b6dbafa1c4c44ac42f1d2293e3125 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:58:57.755460 6169 catalog_manager.cc:5484] T df00e37199e944589a0a5f3feeee308a P 8315d22c56f7439a9e1fc82f153f018c reported cstate change: term changed from 0 to 1, leader changed from <none> to 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130). New cstate: current_term: 1 leader_uuid: "8315d22c56f7439a9e1fc82f153f018c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } health_report { overall_health: UNKNOWN } } }
I20240501 07:58:57.755511 6164 catalog_manager.cc:5484] T 114b6dbafa1c4c44ac42f1d2293e3125 P 3b164cbbc00f4e8eb3254ddd349a3ff9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } health_report { overall_health: UNKNOWN } } }
I20240501 07:58:57.795797 7583 raft_consensus.cc:482] T f68dbc1f65eb4f1b91a62e2de82a7676 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:57.796604 7583 raft_consensus.cc:504] T f68dbc1f65eb4f1b91a62e2de82a7676 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } }
I20240501 07:58:57.797139 7583 leader_election.cc:288] T f68dbc1f65eb4f1b91a62e2de82a7676 P 8b8b3ad3aa8f4053b2167dd550f2086f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130:32941), 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131:34471)
I20240501 07:58:57.797624 6577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f68dbc1f65eb4f1b91a62e2de82a7676" candidate_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" is_pre_election: true
I20240501 07:58:57.797626 6445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f68dbc1f65eb4f1b91a62e2de82a7676" candidate_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8315d22c56f7439a9e1fc82f153f018c" is_pre_election: true
I20240501 07:58:57.797785 6445 raft_consensus.cc:2454] T f68dbc1f65eb4f1b91a62e2de82a7676 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b8b3ad3aa8f4053b2167dd550f2086f in term 0.
I20240501 07:58:57.797780 6577 raft_consensus.cc:2454] T f68dbc1f65eb4f1b91a62e2de82a7676 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b8b3ad3aa8f4053b2167dd550f2086f in term 0.
I20240501 07:58:57.798121 6250 leader_election.cc:302] T f68dbc1f65eb4f1b91a62e2de82a7676 P 8b8b3ad3aa8f4053b2167dd550f2086f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8315d22c56f7439a9e1fc82f153f018c, 8b8b3ad3aa8f4053b2167dd550f2086f; no voters:
I20240501 07:58:57.798292 7583 raft_consensus.cc:2781] T f68dbc1f65eb4f1b91a62e2de82a7676 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:57.798384 7583 raft_consensus.cc:482] T f68dbc1f65eb4f1b91a62e2de82a7676 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:57.798429 7583 raft_consensus.cc:3037] T f68dbc1f65eb4f1b91a62e2de82a7676 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:57.850769 6446 raft_consensus.cc:2454] T 114b6dbafa1c4c44ac42f1d2293e3125 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b164cbbc00f4e8eb3254ddd349a3ff9 in term 1.
I20240501 07:58:57.850775 7583 raft_consensus.cc:504] T f68dbc1f65eb4f1b91a62e2de82a7676 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } }
I20240501 07:58:57.851282 7583 leader_election.cc:288] T f68dbc1f65eb4f1b91a62e2de82a7676 P 8b8b3ad3aa8f4053b2167dd550f2086f [CANDIDATE]: Term 1 election: Requested vote from peers 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130:32941), 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131:34471)
I20240501 07:58:57.851668 6446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f68dbc1f65eb4f1b91a62e2de82a7676" candidate_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8315d22c56f7439a9e1fc82f153f018c"
I20240501 07:58:57.851672 6577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f68dbc1f65eb4f1b91a62e2de82a7676" candidate_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9"
I20240501 07:58:57.851802 6577 raft_consensus.cc:3037] T f68dbc1f65eb4f1b91a62e2de82a7676 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:57.851804 6446 raft_consensus.cc:3037] T f68dbc1f65eb4f1b91a62e2de82a7676 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:57.852792 6446 raft_consensus.cc:2454] T f68dbc1f65eb4f1b91a62e2de82a7676 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b8b3ad3aa8f4053b2167dd550f2086f in term 1.
I20240501 07:58:57.852792 6577 raft_consensus.cc:2454] T f68dbc1f65eb4f1b91a62e2de82a7676 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b8b3ad3aa8f4053b2167dd550f2086f in term 1.
I20240501 07:58:57.853150 6251 leader_election.cc:302] T f68dbc1f65eb4f1b91a62e2de82a7676 P 8b8b3ad3aa8f4053b2167dd550f2086f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b164cbbc00f4e8eb3254ddd349a3ff9, 8b8b3ad3aa8f4053b2167dd550f2086f; no voters:
I20240501 07:58:57.853348 7583 raft_consensus.cc:2781] T f68dbc1f65eb4f1b91a62e2de82a7676 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:57.853456 7583 raft_consensus.cc:686] T f68dbc1f65eb4f1b91a62e2de82a7676 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 LEADER]: Becoming Leader. State: Replica: 8b8b3ad3aa8f4053b2167dd550f2086f, State: Running, Role: LEADER
I20240501 07:58:57.853636 7583 consensus_queue.cc:237] T f68dbc1f65eb4f1b91a62e2de82a7676 P 8b8b3ad3aa8f4053b2167dd550f2086f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } }
I20240501 07:58:57.855573 6169 catalog_manager.cc:5484] T f68dbc1f65eb4f1b91a62e2de82a7676 P 8b8b3ad3aa8f4053b2167dd550f2086f reported cstate change: term changed from 0 to 1, leader changed from <none> to 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129). New cstate: current_term: 1 leader_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } health_report { overall_health: UNKNOWN } } }
W20240501 07:58:57.993136 7696 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:58:57.993425 7696 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:58:57.996779 7696 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_5'
I20240501 07:58:58.098263 6707 raft_consensus.cc:1261] T 114b6dbafa1c4c44ac42f1d2293e3125 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Refusing update from remote peer 3b164cbbc00f4e8eb3254ddd349a3ff9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:58.098618 6446 raft_consensus.cc:1261] T 114b6dbafa1c4c44ac42f1d2293e3125 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Refusing update from remote peer 3b164cbbc00f4e8eb3254ddd349a3ff9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:58.098848 6577 raft_consensus.cc:1261] T df00e37199e944589a0a5f3feeee308a P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Refusing update from remote peer 8315d22c56f7439a9e1fc82f153f018c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:58.098985 7618 consensus_queue.cc:1035] T 114b6dbafa1c4c44ac42f1d2293e3125 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:58.099459 6707 raft_consensus.cc:1261] T df00e37199e944589a0a5f3feeee308a P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Refusing update from remote peer 8315d22c56f7439a9e1fc82f153f018c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:58.099690 7321 consensus_queue.cc:1035] T 114b6dbafa1c4c44ac42f1d2293e3125 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:58.099884 7472 consensus_queue.cc:1035] T df00e37199e944589a0a5f3feeee308a P 8315d22c56f7439a9e1fc82f153f018c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:58.100919 6577 raft_consensus.cc:1261] T f68dbc1f65eb4f1b91a62e2de82a7676 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Refusing update from remote peer 8b8b3ad3aa8f4053b2167dd550f2086f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:58.101261 7583 consensus_queue.cc:1035] T f68dbc1f65eb4f1b91a62e2de82a7676 P 8b8b3ad3aa8f4053b2167dd550f2086f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:58.101424 7433 consensus_queue.cc:1035] T df00e37199e944589a0a5f3feeee308a P 8315d22c56f7439a9e1fc82f153f018c [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:58.103489 6445 raft_consensus.cc:1261] T f68dbc1f65eb4f1b91a62e2de82a7676 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Refusing update from remote peer 8b8b3ad3aa8f4053b2167dd550f2086f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:58.105104 7583 consensus_queue.cc:1035] T f68dbc1f65eb4f1b91a62e2de82a7676 P 8b8b3ad3aa8f4053b2167dd550f2086f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 124.348 ms
time per row: 0.0621741 ms
I20240501 07:58:58.167183 6170 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55950:
name: "rebalance_test_table_6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 07:58:58.172648 6295 tablet_service.cc:1467] Processing CreateTablet for tablet 26713549654249b09f820a31ee19e2b7 (DEFAULT_TABLE table=rebalance_test_table_6 [id=09bfb3350ac042c892c1ef5bc6adf5e2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:58.172679 6687 tablet_service.cc:1467] Processing CreateTablet for tablet 26713549654249b09f820a31ee19e2b7 (DEFAULT_TABLE table=rebalance_test_table_6 [id=09bfb3350ac042c892c1ef5bc6adf5e2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:58.172678 6426 tablet_service.cc:1467] Processing CreateTablet for tablet 26713549654249b09f820a31ee19e2b7 (DEFAULT_TABLE table=rebalance_test_table_6 [id=09bfb3350ac042c892c1ef5bc6adf5e2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:58.173020 6426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26713549654249b09f820a31ee19e2b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:58.173020 6687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26713549654249b09f820a31ee19e2b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:58.173280 6295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26713549654249b09f820a31ee19e2b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:58.173630 6556 tablet_service.cc:1467] Processing CreateTablet for tablet ec2d3ccf0937430f947f3e2bde0506c5 (DEFAULT_TABLE table=rebalance_test_table_6 [id=09bfb3350ac042c892c1ef5bc6adf5e2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:58.173877 6556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec2d3ccf0937430f947f3e2bde0506c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:58.174053 6688 tablet_service.cc:1467] Processing CreateTablet for tablet ec2d3ccf0937430f947f3e2bde0506c5 (DEFAULT_TABLE table=rebalance_test_table_6 [id=09bfb3350ac042c892c1ef5bc6adf5e2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:58.174268 6688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec2d3ccf0937430f947f3e2bde0506c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:58.174908 6424 tablet_service.cc:1467] Processing CreateTablet for tablet e4c392558af14097b92f4cb35978eebe (DEFAULT_TABLE table=rebalance_test_table_6 [id=09bfb3350ac042c892c1ef5bc6adf5e2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:58.175127 6424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4c392558af14097b92f4cb35978eebe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:58.175904 6295 tablet_service.cc:1467] Processing CreateTablet for tablet ec2d3ccf0937430f947f3e2bde0506c5 (DEFAULT_TABLE table=rebalance_test_table_6 [id=09bfb3350ac042c892c1ef5bc6adf5e2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:58.175983 6294 tablet_service.cc:1467] Processing CreateTablet for tablet e4c392558af14097b92f4cb35978eebe (DEFAULT_TABLE table=rebalance_test_table_6 [id=09bfb3350ac042c892c1ef5bc6adf5e2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:58.176127 6295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec2d3ccf0937430f947f3e2bde0506c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:58.176237 6686 tablet_service.cc:1467] Processing CreateTablet for tablet e4c392558af14097b92f4cb35978eebe (DEFAULT_TABLE table=rebalance_test_table_6 [id=09bfb3350ac042c892c1ef5bc6adf5e2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:58.176453 6294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4c392558af14097b92f4cb35978eebe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:58.176465 6686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4c392558af14097b92f4cb35978eebe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:58.178009 7763 tablet_bootstrap.cc:492] T ec2d3ccf0937430f947f3e2bde0506c5 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Bootstrap starting.
I20240501 07:58:58.178416 7762 tablet_bootstrap.cc:492] T 26713549654249b09f820a31ee19e2b7 P 8b8b3ad3aa8f4053b2167dd550f2086f: Bootstrap starting.
I20240501 07:58:58.178998 7763 tablet_bootstrap.cc:654] T ec2d3ccf0937430f947f3e2bde0506c5 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:58.179065 7761 tablet_bootstrap.cc:492] T 26713549654249b09f820a31ee19e2b7 P 8315d22c56f7439a9e1fc82f153f018c: Bootstrap starting.
I20240501 07:58:58.179314 7762 tablet_bootstrap.cc:654] T 26713549654249b09f820a31ee19e2b7 P 8b8b3ad3aa8f4053b2167dd550f2086f: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:58.179953 7761 tablet_bootstrap.cc:654] T 26713549654249b09f820a31ee19e2b7 P 8315d22c56f7439a9e1fc82f153f018c: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:58.180159 7763 tablet_bootstrap.cc:492] T ec2d3ccf0937430f947f3e2bde0506c5 P 3b164cbbc00f4e8eb3254ddd349a3ff9: No bootstrap required, opened a new log
I20240501 07:58:58.180243 7763 ts_tablet_manager.cc:1367] T ec2d3ccf0937430f947f3e2bde0506c5 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 07:58:58.180399 7762 tablet_bootstrap.cc:492] T 26713549654249b09f820a31ee19e2b7 P 8b8b3ad3aa8f4053b2167dd550f2086f: No bootstrap required, opened a new log
I20240501 07:58:58.180485 7762 ts_tablet_manager.cc:1367] T 26713549654249b09f820a31ee19e2b7 P 8b8b3ad3aa8f4053b2167dd550f2086f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:58:58.180723 7763 raft_consensus.cc:348] T ec2d3ccf0937430f947f3e2bde0506c5 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } }
I20240501 07:58:58.180837 7763 raft_consensus.cc:374] T ec2d3ccf0937430f947f3e2bde0506c5 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:58.180871 7763 raft_consensus.cc:729] T ec2d3ccf0937430f947f3e2bde0506c5 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b164cbbc00f4e8eb3254ddd349a3ff9, State: Initialized, Role: FOLLOWER
I20240501 07:58:58.180933 7761 tablet_bootstrap.cc:492] T 26713549654249b09f820a31ee19e2b7 P 8315d22c56f7439a9e1fc82f153f018c: No bootstrap required, opened a new log
I20240501 07:58:58.180948 7762 raft_consensus.cc:348] T 26713549654249b09f820a31ee19e2b7 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } }
I20240501 07:58:58.181051 7761 ts_tablet_manager.cc:1367] T 26713549654249b09f820a31ee19e2b7 P 8315d22c56f7439a9e1fc82f153f018c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 07:58:58.181013 7763 consensus_queue.cc:260] T ec2d3ccf0937430f947f3e2bde0506c5 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } }
I20240501 07:58:58.181068 7762 raft_consensus.cc:374] T 26713549654249b09f820a31ee19e2b7 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:58.181099 7762 raft_consensus.cc:729] T 26713549654249b09f820a31ee19e2b7 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b8b3ad3aa8f4053b2167dd550f2086f, State: Initialized, Role: FOLLOWER
I20240501 07:58:58.181214 7762 consensus_queue.cc:260] T 26713549654249b09f820a31ee19e2b7 P 8b8b3ad3aa8f4053b2167dd550f2086f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } }
I20240501 07:58:58.181314 7763 ts_tablet_manager.cc:1398] T ec2d3ccf0937430f947f3e2bde0506c5 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:58.181461 7762 ts_tablet_manager.cc:1398] T 26713549654249b09f820a31ee19e2b7 P 8b8b3ad3aa8f4053b2167dd550f2086f: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 07:58:58.181460 7761 raft_consensus.cc:348] T 26713549654249b09f820a31ee19e2b7 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } }
I20240501 07:58:58.181555 7761 raft_consensus.cc:374] T 26713549654249b09f820a31ee19e2b7 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:58.181588 7761 raft_consensus.cc:729] T 26713549654249b09f820a31ee19e2b7 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8315d22c56f7439a9e1fc82f153f018c, State: Initialized, Role: FOLLOWER
I20240501 07:58:58.181644 7762 tablet_bootstrap.cc:492] T ec2d3ccf0937430f947f3e2bde0506c5 P 8b8b3ad3aa8f4053b2167dd550f2086f: Bootstrap starting.
I20240501 07:58:58.181704 7761 consensus_queue.cc:260] T 26713549654249b09f820a31ee19e2b7 P 8315d22c56f7439a9e1fc82f153f018c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } }
I20240501 07:58:58.181979 7761 ts_tablet_manager.cc:1398] T 26713549654249b09f820a31ee19e2b7 P 8315d22c56f7439a9e1fc82f153f018c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:58.182152 7761 tablet_bootstrap.cc:492] T e4c392558af14097b92f4cb35978eebe P 8315d22c56f7439a9e1fc82f153f018c: Bootstrap starting.
I20240501 07:58:58.182461 7762 tablet_bootstrap.cc:654] T ec2d3ccf0937430f947f3e2bde0506c5 P 8b8b3ad3aa8f4053b2167dd550f2086f: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:58.183055 7761 tablet_bootstrap.cc:654] T e4c392558af14097b92f4cb35978eebe P 8315d22c56f7439a9e1fc82f153f018c: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:58.184064 7761 tablet_bootstrap.cc:492] T e4c392558af14097b92f4cb35978eebe P 8315d22c56f7439a9e1fc82f153f018c: No bootstrap required, opened a new log
I20240501 07:58:58.184094 7762 tablet_bootstrap.cc:492] T ec2d3ccf0937430f947f3e2bde0506c5 P 8b8b3ad3aa8f4053b2167dd550f2086f: No bootstrap required, opened a new log
I20240501 07:58:58.184146 7761 ts_tablet_manager.cc:1367] T e4c392558af14097b92f4cb35978eebe P 8315d22c56f7439a9e1fc82f153f018c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 07:58:58.184154 7762 ts_tablet_manager.cc:1367] T ec2d3ccf0937430f947f3e2bde0506c5 P 8b8b3ad3aa8f4053b2167dd550f2086f: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20240501 07:58:58.184561 7761 raft_consensus.cc:348] T e4c392558af14097b92f4cb35978eebe P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } }
I20240501 07:58:58.184561 7762 raft_consensus.cc:348] T ec2d3ccf0937430f947f3e2bde0506c5 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } }
I20240501 07:58:58.184659 7761 raft_consensus.cc:374] T e4c392558af14097b92f4cb35978eebe P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:58.184659 7762 raft_consensus.cc:374] T ec2d3ccf0937430f947f3e2bde0506c5 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:58.184692 7762 raft_consensus.cc:729] T ec2d3ccf0937430f947f3e2bde0506c5 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b8b3ad3aa8f4053b2167dd550f2086f, State: Initialized, Role: FOLLOWER
I20240501 07:58:58.184692 7761 raft_consensus.cc:729] T e4c392558af14097b92f4cb35978eebe P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8315d22c56f7439a9e1fc82f153f018c, State: Initialized, Role: FOLLOWER
I20240501 07:58:58.184801 7762 consensus_queue.cc:260] T ec2d3ccf0937430f947f3e2bde0506c5 P 8b8b3ad3aa8f4053b2167dd550f2086f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } }
I20240501 07:58:58.184801 7761 consensus_queue.cc:260] T e4c392558af14097b92f4cb35978eebe P 8315d22c56f7439a9e1fc82f153f018c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } }
I20240501 07:58:58.185077 7761 ts_tablet_manager.cc:1398] T e4c392558af14097b92f4cb35978eebe P 8315d22c56f7439a9e1fc82f153f018c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:58.185077 7762 ts_tablet_manager.cc:1398] T ec2d3ccf0937430f947f3e2bde0506c5 P 8b8b3ad3aa8f4053b2167dd550f2086f: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 07:58:58.185223 7762 tablet_bootstrap.cc:492] T e4c392558af14097b92f4cb35978eebe P 8b8b3ad3aa8f4053b2167dd550f2086f: Bootstrap starting.
I20240501 07:58:58.186074 7762 tablet_bootstrap.cc:654] T e4c392558af14097b92f4cb35978eebe P 8b8b3ad3aa8f4053b2167dd550f2086f: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:58.187419 7762 tablet_bootstrap.cc:492] T e4c392558af14097b92f4cb35978eebe P 8b8b3ad3aa8f4053b2167dd550f2086f: No bootstrap required, opened a new log
I20240501 07:58:58.187507 7762 ts_tablet_manager.cc:1367] T e4c392558af14097b92f4cb35978eebe P 8b8b3ad3aa8f4053b2167dd550f2086f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 07:58:58.187914 7762 raft_consensus.cc:348] T e4c392558af14097b92f4cb35978eebe P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } }
I20240501 07:58:58.187995 7762 raft_consensus.cc:374] T e4c392558af14097b92f4cb35978eebe P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:58.188016 7762 raft_consensus.cc:729] T e4c392558af14097b92f4cb35978eebe P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b8b3ad3aa8f4053b2167dd550f2086f, State: Initialized, Role: FOLLOWER
I20240501 07:58:58.188115 7762 consensus_queue.cc:260] T e4c392558af14097b92f4cb35978eebe P 8b8b3ad3aa8f4053b2167dd550f2086f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } }
I20240501 07:58:58.188345 7762 ts_tablet_manager.cc:1398] T e4c392558af14097b92f4cb35978eebe P 8b8b3ad3aa8f4053b2167dd550f2086f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:58:58.191381 7770 tablet_bootstrap.cc:492] T 26713549654249b09f820a31ee19e2b7 P 1f9cb72be98941b5a8e504cbe34132b9: Bootstrap starting.
I20240501 07:58:58.192243 7770 tablet_bootstrap.cc:654] T 26713549654249b09f820a31ee19e2b7 P 1f9cb72be98941b5a8e504cbe34132b9: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:58.193298 7770 tablet_bootstrap.cc:492] T 26713549654249b09f820a31ee19e2b7 P 1f9cb72be98941b5a8e504cbe34132b9: No bootstrap required, opened a new log
I20240501 07:58:58.193372 7770 ts_tablet_manager.cc:1367] T 26713549654249b09f820a31ee19e2b7 P 1f9cb72be98941b5a8e504cbe34132b9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:58:58.193743 7770 raft_consensus.cc:348] T 26713549654249b09f820a31ee19e2b7 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } }
I20240501 07:58:58.193831 7770 raft_consensus.cc:374] T 26713549654249b09f820a31ee19e2b7 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:58.193852 7770 raft_consensus.cc:729] T 26713549654249b09f820a31ee19e2b7 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f9cb72be98941b5a8e504cbe34132b9, State: Initialized, Role: FOLLOWER
I20240501 07:58:58.193962 7770 consensus_queue.cc:260] T 26713549654249b09f820a31ee19e2b7 P 1f9cb72be98941b5a8e504cbe34132b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } }
I20240501 07:58:58.194216 7770 ts_tablet_manager.cc:1398] T 26713549654249b09f820a31ee19e2b7 P 1f9cb72be98941b5a8e504cbe34132b9: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:58:58.194466 7770 tablet_bootstrap.cc:492] T e4c392558af14097b92f4cb35978eebe P 1f9cb72be98941b5a8e504cbe34132b9: Bootstrap starting.
I20240501 07:58:58.195449 7770 tablet_bootstrap.cc:654] T e4c392558af14097b92f4cb35978eebe P 1f9cb72be98941b5a8e504cbe34132b9: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:58.196532 7770 tablet_bootstrap.cc:492] T e4c392558af14097b92f4cb35978eebe P 1f9cb72be98941b5a8e504cbe34132b9: No bootstrap required, opened a new log
I20240501 07:58:58.196758 7770 ts_tablet_manager.cc:1367] T e4c392558af14097b92f4cb35978eebe P 1f9cb72be98941b5a8e504cbe34132b9: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 07:58:58.197271 7770 raft_consensus.cc:348] T e4c392558af14097b92f4cb35978eebe P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } }
I20240501 07:58:58.197399 7770 raft_consensus.cc:374] T e4c392558af14097b92f4cb35978eebe P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:58.197443 7770 raft_consensus.cc:729] T e4c392558af14097b92f4cb35978eebe P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f9cb72be98941b5a8e504cbe34132b9, State: Initialized, Role: FOLLOWER
I20240501 07:58:58.197572 7770 consensus_queue.cc:260] T e4c392558af14097b92f4cb35978eebe P 1f9cb72be98941b5a8e504cbe34132b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } }
I20240501 07:58:58.197888 7770 ts_tablet_manager.cc:1398] T e4c392558af14097b92f4cb35978eebe P 1f9cb72be98941b5a8e504cbe34132b9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:58.198029 7770 tablet_bootstrap.cc:492] T ec2d3ccf0937430f947f3e2bde0506c5 P 1f9cb72be98941b5a8e504cbe34132b9: Bootstrap starting.
I20240501 07:58:58.199163 7770 tablet_bootstrap.cc:654] T ec2d3ccf0937430f947f3e2bde0506c5 P 1f9cb72be98941b5a8e504cbe34132b9: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:58.200053 7770 tablet_bootstrap.cc:492] T ec2d3ccf0937430f947f3e2bde0506c5 P 1f9cb72be98941b5a8e504cbe34132b9: No bootstrap required, opened a new log
I20240501 07:58:58.200122 7770 ts_tablet_manager.cc:1367] T ec2d3ccf0937430f947f3e2bde0506c5 P 1f9cb72be98941b5a8e504cbe34132b9: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 07:58:58.200510 7770 raft_consensus.cc:348] T ec2d3ccf0937430f947f3e2bde0506c5 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } }
I20240501 07:58:58.200592 7770 raft_consensus.cc:374] T ec2d3ccf0937430f947f3e2bde0506c5 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:58.200614 7770 raft_consensus.cc:729] T ec2d3ccf0937430f947f3e2bde0506c5 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f9cb72be98941b5a8e504cbe34132b9, State: Initialized, Role: FOLLOWER
I20240501 07:58:58.200724 7770 consensus_queue.cc:260] T ec2d3ccf0937430f947f3e2bde0506c5 P 1f9cb72be98941b5a8e504cbe34132b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } }
I20240501 07:58:58.201057 7770 ts_tablet_manager.cc:1398] T ec2d3ccf0937430f947f3e2bde0506c5 P 1f9cb72be98941b5a8e504cbe34132b9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:58.237761 7467 raft_consensus.cc:482] T e4c392558af14097b92f4cb35978eebe P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:58.237907 7467 raft_consensus.cc:504] T e4c392558af14097b92f4cb35978eebe P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } }
I20240501 07:58:58.238369 7467 leader_election.cc:288] T e4c392558af14097b92f4cb35978eebe P 1f9cb72be98941b5a8e504cbe34132b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129:46875), 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130:32941)
I20240501 07:58:58.238678 6314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4c392558af14097b92f4cb35978eebe" candidate_uuid: "1f9cb72be98941b5a8e504cbe34132b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" is_pre_election: true
I20240501 07:58:58.238762 6446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4c392558af14097b92f4cb35978eebe" candidate_uuid: "1f9cb72be98941b5a8e504cbe34132b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8315d22c56f7439a9e1fc82f153f018c" is_pre_election: true
I20240501 07:58:58.238816 6314 raft_consensus.cc:2454] T e4c392558af14097b92f4cb35978eebe P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f9cb72be98941b5a8e504cbe34132b9 in term 0.
I20240501 07:58:58.238907 6446 raft_consensus.cc:2454] T e4c392558af14097b92f4cb35978eebe P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f9cb72be98941b5a8e504cbe34132b9 in term 0.
I20240501 07:58:58.239125 6641 leader_election.cc:302] T e4c392558af14097b92f4cb35978eebe P 1f9cb72be98941b5a8e504cbe34132b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f9cb72be98941b5a8e504cbe34132b9, 8b8b3ad3aa8f4053b2167dd550f2086f; no voters:
I20240501 07:58:58.239284 7467 raft_consensus.cc:2781] T e4c392558af14097b92f4cb35978eebe P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:58.239347 7467 raft_consensus.cc:482] T e4c392558af14097b92f4cb35978eebe P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:58.239387 7467 raft_consensus.cc:3037] T e4c392558af14097b92f4cb35978eebe P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:58.240239 7467 raft_consensus.cc:504] T e4c392558af14097b92f4cb35978eebe P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } }
I20240501 07:58:58.240550 7467 leader_election.cc:288] T e4c392558af14097b92f4cb35978eebe P 1f9cb72be98941b5a8e504cbe34132b9 [CANDIDATE]: Term 1 election: Requested vote from peers 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129:46875), 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130:32941)
I20240501 07:58:58.240901 6314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4c392558af14097b92f4cb35978eebe" candidate_uuid: "1f9cb72be98941b5a8e504cbe34132b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f"
I20240501 07:58:58.241027 6314 raft_consensus.cc:3037] T e4c392558af14097b92f4cb35978eebe P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:58.241003 6446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4c392558af14097b92f4cb35978eebe" candidate_uuid: "1f9cb72be98941b5a8e504cbe34132b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8315d22c56f7439a9e1fc82f153f018c"
I20240501 07:58:58.241117 6446 raft_consensus.cc:3037] T e4c392558af14097b92f4cb35978eebe P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:58.241876 6446 raft_consensus.cc:2454] T e4c392558af14097b92f4cb35978eebe P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f9cb72be98941b5a8e504cbe34132b9 in term 1.
I20240501 07:58:58.241894 6314 raft_consensus.cc:2454] T e4c392558af14097b92f4cb35978eebe P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f9cb72be98941b5a8e504cbe34132b9 in term 1.
I20240501 07:58:58.242158 6643 leader_election.cc:302] T e4c392558af14097b92f4cb35978eebe P 1f9cb72be98941b5a8e504cbe34132b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f9cb72be98941b5a8e504cbe34132b9, 8315d22c56f7439a9e1fc82f153f018c; no voters:
I20240501 07:58:58.242343 7467 raft_consensus.cc:2781] T e4c392558af14097b92f4cb35978eebe P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:58.242569 7467 raft_consensus.cc:686] T e4c392558af14097b92f4cb35978eebe P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 LEADER]: Becoming Leader. State: Replica: 1f9cb72be98941b5a8e504cbe34132b9, State: Running, Role: LEADER
I20240501 07:58:58.242728 7467 consensus_queue.cc:237] T e4c392558af14097b92f4cb35978eebe P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } }
I20240501 07:58:58.244360 6164 catalog_manager.cc:5484] T e4c392558af14097b92f4cb35978eebe P 1f9cb72be98941b5a8e504cbe34132b9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132). New cstate: current_term: 1 leader_uuid: "1f9cb72be98941b5a8e504cbe34132b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } health_report { overall_health: UNKNOWN } } }
I20240501 07:58:58.311379 7306 raft_consensus.cc:482] T ec2d3ccf0937430f947f3e2bde0506c5 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:58.311523 7306 raft_consensus.cc:504] T ec2d3ccf0937430f947f3e2bde0506c5 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } }
I20240501 07:58:58.311913 7306 leader_election.cc:288] T ec2d3ccf0937430f947f3e2bde0506c5 P 8b8b3ad3aa8f4053b2167dd550f2086f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132:36773), 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131:34471)
I20240501 07:58:58.312381 6576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec2d3ccf0937430f947f3e2bde0506c5" candidate_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" is_pre_election: true
I20240501 07:58:58.312381 6708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec2d3ccf0937430f947f3e2bde0506c5" candidate_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f9cb72be98941b5a8e504cbe34132b9" is_pre_election: true
I20240501 07:58:58.312536 6576 raft_consensus.cc:2454] T ec2d3ccf0937430f947f3e2bde0506c5 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b8b3ad3aa8f4053b2167dd550f2086f in term 0.
I20240501 07:58:58.312536 6708 raft_consensus.cc:2454] T ec2d3ccf0937430f947f3e2bde0506c5 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b8b3ad3aa8f4053b2167dd550f2086f in term 0.
I20240501 07:58:58.312876 6250 leader_election.cc:302] T ec2d3ccf0937430f947f3e2bde0506c5 P 8b8b3ad3aa8f4053b2167dd550f2086f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f9cb72be98941b5a8e504cbe34132b9, 8b8b3ad3aa8f4053b2167dd550f2086f; no voters:
I20240501 07:58:58.313072 7306 raft_consensus.cc:2781] T ec2d3ccf0937430f947f3e2bde0506c5 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:58.313149 7306 raft_consensus.cc:482] T ec2d3ccf0937430f947f3e2bde0506c5 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:58.313179 7306 raft_consensus.cc:3037] T ec2d3ccf0937430f947f3e2bde0506c5 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:58.314146 7306 raft_consensus.cc:504] T ec2d3ccf0937430f947f3e2bde0506c5 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } }
I20240501 07:58:58.314481 7306 leader_election.cc:288] T ec2d3ccf0937430f947f3e2bde0506c5 P 8b8b3ad3aa8f4053b2167dd550f2086f [CANDIDATE]: Term 1 election: Requested vote from peers 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132:36773), 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131:34471)
I20240501 07:58:58.314858 6576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec2d3ccf0937430f947f3e2bde0506c5" candidate_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9"
I20240501 07:58:58.314872 6708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec2d3ccf0937430f947f3e2bde0506c5" candidate_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f9cb72be98941b5a8e504cbe34132b9"
I20240501 07:58:58.314991 6708 raft_consensus.cc:3037] T ec2d3ccf0937430f947f3e2bde0506c5 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:58.314991 6576 raft_consensus.cc:3037] T ec2d3ccf0937430f947f3e2bde0506c5 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:58.315761 6708 raft_consensus.cc:2454] T ec2d3ccf0937430f947f3e2bde0506c5 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b8b3ad3aa8f4053b2167dd550f2086f in term 1.
I20240501 07:58:58.315850 6576 raft_consensus.cc:2454] T ec2d3ccf0937430f947f3e2bde0506c5 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b8b3ad3aa8f4053b2167dd550f2086f in term 1.
I20240501 07:58:58.316035 6250 leader_election.cc:302] T ec2d3ccf0937430f947f3e2bde0506c5 P 8b8b3ad3aa8f4053b2167dd550f2086f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f9cb72be98941b5a8e504cbe34132b9, 8b8b3ad3aa8f4053b2167dd550f2086f; no voters:
I20240501 07:58:58.316277 7306 raft_consensus.cc:2781] T ec2d3ccf0937430f947f3e2bde0506c5 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:58.316345 7306 raft_consensus.cc:686] T ec2d3ccf0937430f947f3e2bde0506c5 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 LEADER]: Becoming Leader. State: Replica: 8b8b3ad3aa8f4053b2167dd550f2086f, State: Running, Role: LEADER
I20240501 07:58:58.316483 7306 consensus_queue.cc:237] T ec2d3ccf0937430f947f3e2bde0506c5 P 8b8b3ad3aa8f4053b2167dd550f2086f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } }
I20240501 07:58:58.318310 6164 catalog_manager.cc:5484] T ec2d3ccf0937430f947f3e2bde0506c5 P 8b8b3ad3aa8f4053b2167dd550f2086f reported cstate change: term changed from 0 to 1, leader changed from <none> to 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129). New cstate: current_term: 1 leader_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } health_report { overall_health: UNKNOWN } } }
I20240501 07:58:58.504176 7774 raft_consensus.cc:482] T 26713549654249b09f820a31ee19e2b7 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:58.504323 7774 raft_consensus.cc:504] T 26713549654249b09f820a31ee19e2b7 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } }
I20240501 07:58:58.504751 7774 leader_election.cc:288] T 26713549654249b09f820a31ee19e2b7 P 1f9cb72be98941b5a8e504cbe34132b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129:46875), 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130:32941)
I20240501 07:58:58.505110 6446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26713549654249b09f820a31ee19e2b7" candidate_uuid: "1f9cb72be98941b5a8e504cbe34132b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8315d22c56f7439a9e1fc82f153f018c" is_pre_election: true
I20240501 07:58:58.505192 6314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26713549654249b09f820a31ee19e2b7" candidate_uuid: "1f9cb72be98941b5a8e504cbe34132b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" is_pre_election: true
I20240501 07:58:58.505277 6446 raft_consensus.cc:2454] T 26713549654249b09f820a31ee19e2b7 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f9cb72be98941b5a8e504cbe34132b9 in term 0.
I20240501 07:58:58.505353 6314 raft_consensus.cc:2454] T 26713549654249b09f820a31ee19e2b7 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f9cb72be98941b5a8e504cbe34132b9 in term 0.
I20240501 07:58:58.505587 6643 leader_election.cc:302] T 26713549654249b09f820a31ee19e2b7 P 1f9cb72be98941b5a8e504cbe34132b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f9cb72be98941b5a8e504cbe34132b9, 8315d22c56f7439a9e1fc82f153f018c; no voters:
I20240501 07:58:58.505746 7774 raft_consensus.cc:2781] T 26713549654249b09f820a31ee19e2b7 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:58.505805 7774 raft_consensus.cc:482] T 26713549654249b09f820a31ee19e2b7 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:58.505846 7774 raft_consensus.cc:3037] T 26713549654249b09f820a31ee19e2b7 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:58.506752 7774 raft_consensus.cc:504] T 26713549654249b09f820a31ee19e2b7 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } }
I20240501 07:58:58.507061 7774 leader_election.cc:288] T 26713549654249b09f820a31ee19e2b7 P 1f9cb72be98941b5a8e504cbe34132b9 [CANDIDATE]: Term 1 election: Requested vote from peers 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129:46875), 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130:32941)
I20240501 07:58:58.507386 6314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26713549654249b09f820a31ee19e2b7" candidate_uuid: "1f9cb72be98941b5a8e504cbe34132b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f"
I20240501 07:58:58.507402 6446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26713549654249b09f820a31ee19e2b7" candidate_uuid: "1f9cb72be98941b5a8e504cbe34132b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8315d22c56f7439a9e1fc82f153f018c"
I20240501 07:58:58.507484 6314 raft_consensus.cc:3037] T 26713549654249b09f820a31ee19e2b7 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:58.507516 6446 raft_consensus.cc:3037] T 26713549654249b09f820a31ee19e2b7 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:58.508224 6314 raft_consensus.cc:2454] T 26713549654249b09f820a31ee19e2b7 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f9cb72be98941b5a8e504cbe34132b9 in term 1.
I20240501 07:58:58.508366 6446 raft_consensus.cc:2454] T 26713549654249b09f820a31ee19e2b7 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f9cb72be98941b5a8e504cbe34132b9 in term 1.
I20240501 07:58:58.508530 6641 leader_election.cc:302] T 26713549654249b09f820a31ee19e2b7 P 1f9cb72be98941b5a8e504cbe34132b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f9cb72be98941b5a8e504cbe34132b9, 8b8b3ad3aa8f4053b2167dd550f2086f; no voters:
I20240501 07:58:58.508806 7774 raft_consensus.cc:2781] T 26713549654249b09f820a31ee19e2b7 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:58.508891 7774 raft_consensus.cc:686] T 26713549654249b09f820a31ee19e2b7 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 LEADER]: Becoming Leader. State: Replica: 1f9cb72be98941b5a8e504cbe34132b9, State: Running, Role: LEADER
I20240501 07:58:58.509056 7774 consensus_queue.cc:237] T 26713549654249b09f820a31ee19e2b7 P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } }
I20240501 07:58:58.510857 6164 catalog_manager.cc:5484] T 26713549654249b09f820a31ee19e2b7 P 1f9cb72be98941b5a8e504cbe34132b9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132). New cstate: current_term: 1 leader_uuid: "1f9cb72be98941b5a8e504cbe34132b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } health_report { overall_health: UNKNOWN } } }
W20240501 07:58:58.649670 7778 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:58:58.649958 7778 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:58:58.653415 7778 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_6'
I20240501 07:58:58.731171 6576 raft_consensus.cc:1261] T ec2d3ccf0937430f947f3e2bde0506c5 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Refusing update from remote peer 8b8b3ad3aa8f4053b2167dd550f2086f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:58.731649 7583 consensus_queue.cc:1035] T ec2d3ccf0937430f947f3e2bde0506c5 P 8b8b3ad3aa8f4053b2167dd550f2086f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:58.732378 6708 raft_consensus.cc:1261] T ec2d3ccf0937430f947f3e2bde0506c5 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Refusing update from remote peer 8b8b3ad3aa8f4053b2167dd550f2086f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:58.732863 7583 consensus_queue.cc:1035] T ec2d3ccf0937430f947f3e2bde0506c5 P 8b8b3ad3aa8f4053b2167dd550f2086f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:58.738039 6446 raft_consensus.cc:1261] T e4c392558af14097b92f4cb35978eebe P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Refusing update from remote peer 1f9cb72be98941b5a8e504cbe34132b9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:58.738772 7774 consensus_queue.cc:1035] T e4c392558af14097b92f4cb35978eebe P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:58.744465 6314 raft_consensus.cc:1261] T 26713549654249b09f820a31ee19e2b7 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Refusing update from remote peer 1f9cb72be98941b5a8e504cbe34132b9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:58.745256 6446 raft_consensus.cc:1261] T 26713549654249b09f820a31ee19e2b7 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Refusing update from remote peer 1f9cb72be98941b5a8e504cbe34132b9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:58.745229 7822 consensus_queue.cc:1035] T 26713549654249b09f820a31ee19e2b7 P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:58.745734 7822 consensus_queue.cc:1035] T 26713549654249b09f820a31ee19e2b7 P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:58.749992 6315 raft_consensus.cc:1261] T e4c392558af14097b92f4cb35978eebe P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Refusing update from remote peer 1f9cb72be98941b5a8e504cbe34132b9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:58.752079 7467 consensus_queue.cc:1035] T e4c392558af14097b92f4cb35978eebe P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:58.766736 7810 mvcc.cc:205] Tried to move back new op lower bound from 7022798187467296768 to 7022798186534658048. Current Snapshot: MvccSnapshot[applied={T|T < 7022798187467296768}]
I20240501 07:58:58.781586 7809 mvcc.cc:205] Tried to move back new op lower bound from 7022798187467296768 to 7022798186534658048. Current Snapshot: MvccSnapshot[applied={T|T < 7022798187467296768}]
I20240501 07:58:58.784054 7818 mvcc.cc:205] Tried to move back new op lower bound from 7022798187467296768 to 7022798186534658048. Current Snapshot: MvccSnapshot[applied={T|T < 7022798187467296768}]
INSERT report
rows total: 2000
time total: 105.317 ms
time per row: 0.0526583 ms
I20240501 07:58:58.804399 6164 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55950:
name: "rebalance_test_table_7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 07:58:58.810075 6294 tablet_service.cc:1467] Processing CreateTablet for tablet e651f523d85547c0ad834168110cbc6d (DEFAULT_TABLE table=rebalance_test_table_7 [id=8e242b97203b46d0840cb3b6a649e5a7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:58.810269 6556 tablet_service.cc:1467] Processing CreateTablet for tablet e651f523d85547c0ad834168110cbc6d (DEFAULT_TABLE table=rebalance_test_table_7 [id=8e242b97203b46d0840cb3b6a649e5a7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:58.810315 6424 tablet_service.cc:1467] Processing CreateTablet for tablet e651f523d85547c0ad834168110cbc6d (DEFAULT_TABLE table=rebalance_test_table_7 [id=8e242b97203b46d0840cb3b6a649e5a7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:58.810457 6294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e651f523d85547c0ad834168110cbc6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:58.810565 6556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e651f523d85547c0ad834168110cbc6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:58.810592 6424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e651f523d85547c0ad834168110cbc6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:58.812455 7838 tablet_bootstrap.cc:492] T e651f523d85547c0ad834168110cbc6d P 3b164cbbc00f4e8eb3254ddd349a3ff9: Bootstrap starting.
I20240501 07:58:58.812947 6556 tablet_service.cc:1467] Processing CreateTablet for tablet 8454f24b0c28451b8ee51be39ca3e07f (DEFAULT_TABLE table=rebalance_test_table_7 [id=8e242b97203b46d0840cb3b6a649e5a7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:58.813181 6556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8454f24b0c28451b8ee51be39ca3e07f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:58.813407 7838 tablet_bootstrap.cc:654] T e651f523d85547c0ad834168110cbc6d P 3b164cbbc00f4e8eb3254ddd349a3ff9: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:58.813561 6688 tablet_service.cc:1467] Processing CreateTablet for tablet 8454f24b0c28451b8ee51be39ca3e07f (DEFAULT_TABLE table=rebalance_test_table_7 [id=8e242b97203b46d0840cb3b6a649e5a7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:58.813683 6294 tablet_service.cc:1467] Processing CreateTablet for tablet 8454f24b0c28451b8ee51be39ca3e07f (DEFAULT_TABLE table=rebalance_test_table_7 [id=8e242b97203b46d0840cb3b6a649e5a7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:58.813824 6688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8454f24b0c28451b8ee51be39ca3e07f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:58.813918 6294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8454f24b0c28451b8ee51be39ca3e07f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:58.815061 6557 tablet_service.cc:1467] Processing CreateTablet for tablet c3a9036ccfee45498bca9681518d3699 (DEFAULT_TABLE table=rebalance_test_table_7 [id=8e242b97203b46d0840cb3b6a649e5a7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:58.815248 6557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3a9036ccfee45498bca9681518d3699. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:58.815935 7840 tablet_bootstrap.cc:492] T e651f523d85547c0ad834168110cbc6d P 8b8b3ad3aa8f4053b2167dd550f2086f: Bootstrap starting.
I20240501 07:58:58.816167 6687 tablet_service.cc:1467] Processing CreateTablet for tablet c3a9036ccfee45498bca9681518d3699 (DEFAULT_TABLE table=rebalance_test_table_7 [id=8e242b97203b46d0840cb3b6a649e5a7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:58.816385 6687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3a9036ccfee45498bca9681518d3699. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:58.816802 7840 tablet_bootstrap.cc:654] T e651f523d85547c0ad834168110cbc6d P 8b8b3ad3aa8f4053b2167dd550f2086f: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:58.817266 7842 tablet_bootstrap.cc:492] T 8454f24b0c28451b8ee51be39ca3e07f P 1f9cb72be98941b5a8e504cbe34132b9: Bootstrap starting.
I20240501 07:58:58.817670 7839 tablet_bootstrap.cc:492] T e651f523d85547c0ad834168110cbc6d P 8315d22c56f7439a9e1fc82f153f018c: Bootstrap starting.
I20240501 07:58:58.818560 7838 tablet_bootstrap.cc:492] T e651f523d85547c0ad834168110cbc6d P 3b164cbbc00f4e8eb3254ddd349a3ff9: No bootstrap required, opened a new log
I20240501 07:58:58.818696 7838 ts_tablet_manager.cc:1367] T e651f523d85547c0ad834168110cbc6d P 3b164cbbc00f4e8eb3254ddd349a3ff9: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.002s
I20240501 07:58:58.818735 7839 tablet_bootstrap.cc:654] T e651f523d85547c0ad834168110cbc6d P 8315d22c56f7439a9e1fc82f153f018c: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:58.819167 7838 raft_consensus.cc:348] T e651f523d85547c0ad834168110cbc6d P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } }
I20240501 07:58:58.819294 7838 raft_consensus.cc:374] T e651f523d85547c0ad834168110cbc6d P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:58.819337 7838 raft_consensus.cc:729] T e651f523d85547c0ad834168110cbc6d P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b164cbbc00f4e8eb3254ddd349a3ff9, State: Initialized, Role: FOLLOWER
I20240501 07:58:58.819502 7838 consensus_queue.cc:260] T e651f523d85547c0ad834168110cbc6d P 3b164cbbc00f4e8eb3254ddd349a3ff9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } }
I20240501 07:58:58.820008 7838 ts_tablet_manager.cc:1398] T e651f523d85547c0ad834168110cbc6d P 3b164cbbc00f4e8eb3254ddd349a3ff9: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 07:58:58.820240 7838 tablet_bootstrap.cc:492] T 8454f24b0c28451b8ee51be39ca3e07f P 3b164cbbc00f4e8eb3254ddd349a3ff9: Bootstrap starting.
I20240501 07:58:58.820669 6424 tablet_service.cc:1467] Processing CreateTablet for tablet c3a9036ccfee45498bca9681518d3699 (DEFAULT_TABLE table=rebalance_test_table_7 [id=8e242b97203b46d0840cb3b6a649e5a7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:58.820984 6424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3a9036ccfee45498bca9681518d3699. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:58.821564 7838 tablet_bootstrap.cc:654] T 8454f24b0c28451b8ee51be39ca3e07f P 3b164cbbc00f4e8eb3254ddd349a3ff9: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:58.822252 7842 tablet_bootstrap.cc:654] T 8454f24b0c28451b8ee51be39ca3e07f P 1f9cb72be98941b5a8e504cbe34132b9: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:58.823428 7838 tablet_bootstrap.cc:492] T 8454f24b0c28451b8ee51be39ca3e07f P 3b164cbbc00f4e8eb3254ddd349a3ff9: No bootstrap required, opened a new log
I20240501 07:58:58.823578 7838 ts_tablet_manager.cc:1367] T 8454f24b0c28451b8ee51be39ca3e07f P 3b164cbbc00f4e8eb3254ddd349a3ff9: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 07:58:58.824169 7838 raft_consensus.cc:348] T 8454f24b0c28451b8ee51be39ca3e07f P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } }
I20240501 07:58:58.824321 7842 tablet_bootstrap.cc:492] T 8454f24b0c28451b8ee51be39ca3e07f P 1f9cb72be98941b5a8e504cbe34132b9: No bootstrap required, opened a new log
I20240501 07:58:58.824338 7838 raft_consensus.cc:374] T 8454f24b0c28451b8ee51be39ca3e07f P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:58.824416 7842 ts_tablet_manager.cc:1367] T 8454f24b0c28451b8ee51be39ca3e07f P 1f9cb72be98941b5a8e504cbe34132b9: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240501 07:58:58.824497 7838 raft_consensus.cc:729] T 8454f24b0c28451b8ee51be39ca3e07f P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b164cbbc00f4e8eb3254ddd349a3ff9, State: Initialized, Role: FOLLOWER
I20240501 07:58:58.824679 7840 tablet_bootstrap.cc:492] T e651f523d85547c0ad834168110cbc6d P 8b8b3ad3aa8f4053b2167dd550f2086f: No bootstrap required, opened a new log
I20240501 07:58:58.824707 7838 consensus_queue.cc:260] T 8454f24b0c28451b8ee51be39ca3e07f P 3b164cbbc00f4e8eb3254ddd349a3ff9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } }
I20240501 07:58:58.824805 7840 ts_tablet_manager.cc:1367] T e651f523d85547c0ad834168110cbc6d P 8b8b3ad3aa8f4053b2167dd550f2086f: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240501 07:58:58.824826 7842 raft_consensus.cc:348] T 8454f24b0c28451b8ee51be39ca3e07f P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } }
I20240501 07:58:58.825172 7838 ts_tablet_manager.cc:1398] T 8454f24b0c28451b8ee51be39ca3e07f P 3b164cbbc00f4e8eb3254ddd349a3ff9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:58.825254 7842 raft_consensus.cc:374] T 8454f24b0c28451b8ee51be39ca3e07f P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:58.825320 7842 raft_consensus.cc:729] T 8454f24b0c28451b8ee51be39ca3e07f P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f9cb72be98941b5a8e504cbe34132b9, State: Initialized, Role: FOLLOWER
I20240501 07:58:58.825342 7840 raft_consensus.cc:348] T e651f523d85547c0ad834168110cbc6d P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } }
I20240501 07:58:58.825455 7840 raft_consensus.cc:374] T e651f523d85547c0ad834168110cbc6d P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:58.825518 7840 raft_consensus.cc:729] T e651f523d85547c0ad834168110cbc6d P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b8b3ad3aa8f4053b2167dd550f2086f, State: Initialized, Role: FOLLOWER
I20240501 07:58:58.825485 7842 consensus_queue.cc:260] T 8454f24b0c28451b8ee51be39ca3e07f P 1f9cb72be98941b5a8e504cbe34132b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } }
I20240501 07:58:58.825675 7840 consensus_queue.cc:260] T e651f523d85547c0ad834168110cbc6d P 8b8b3ad3aa8f4053b2167dd550f2086f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } }
I20240501 07:58:58.825975 7840 ts_tablet_manager.cc:1398] T e651f523d85547c0ad834168110cbc6d P 8b8b3ad3aa8f4053b2167dd550f2086f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:58.826331 7838 tablet_bootstrap.cc:492] T c3a9036ccfee45498bca9681518d3699 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Bootstrap starting.
I20240501 07:58:58.826601 7840 tablet_bootstrap.cc:492] T 8454f24b0c28451b8ee51be39ca3e07f P 8b8b3ad3aa8f4053b2167dd550f2086f: Bootstrap starting.
I20240501 07:58:58.827566 7840 tablet_bootstrap.cc:654] T 8454f24b0c28451b8ee51be39ca3e07f P 8b8b3ad3aa8f4053b2167dd550f2086f: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:58.827566 7842 ts_tablet_manager.cc:1398] T 8454f24b0c28451b8ee51be39ca3e07f P 1f9cb72be98941b5a8e504cbe34132b9: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 07:58:58.828121 7842 tablet_bootstrap.cc:492] T c3a9036ccfee45498bca9681518d3699 P 1f9cb72be98941b5a8e504cbe34132b9: Bootstrap starting.
I20240501 07:58:58.829008 7842 tablet_bootstrap.cc:654] T c3a9036ccfee45498bca9681518d3699 P 1f9cb72be98941b5a8e504cbe34132b9: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:58.829205 7838 tablet_bootstrap.cc:654] T c3a9036ccfee45498bca9681518d3699 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:58.831110 7840 tablet_bootstrap.cc:492] T 8454f24b0c28451b8ee51be39ca3e07f P 8b8b3ad3aa8f4053b2167dd550f2086f: No bootstrap required, opened a new log
I20240501 07:58:58.831240 7840 ts_tablet_manager.cc:1367] T 8454f24b0c28451b8ee51be39ca3e07f P 8b8b3ad3aa8f4053b2167dd550f2086f: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 07:58:58.831676 7840 raft_consensus.cc:348] T 8454f24b0c28451b8ee51be39ca3e07f P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } }
I20240501 07:58:58.831928 7840 raft_consensus.cc:374] T 8454f24b0c28451b8ee51be39ca3e07f P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:58.831976 7840 raft_consensus.cc:729] T 8454f24b0c28451b8ee51be39ca3e07f P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b8b3ad3aa8f4053b2167dd550f2086f, State: Initialized, Role: FOLLOWER
I20240501 07:58:58.832101 7840 consensus_queue.cc:260] T 8454f24b0c28451b8ee51be39ca3e07f P 8b8b3ad3aa8f4053b2167dd550f2086f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } }
I20240501 07:58:58.832371 7840 ts_tablet_manager.cc:1398] T 8454f24b0c28451b8ee51be39ca3e07f P 8b8b3ad3aa8f4053b2167dd550f2086f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:58.832988 7839 tablet_bootstrap.cc:492] T e651f523d85547c0ad834168110cbc6d P 8315d22c56f7439a9e1fc82f153f018c: No bootstrap required, opened a new log
I20240501 07:58:58.833143 7839 ts_tablet_manager.cc:1367] T e651f523d85547c0ad834168110cbc6d P 8315d22c56f7439a9e1fc82f153f018c: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.000s
I20240501 07:58:58.833699 7839 raft_consensus.cc:348] T e651f523d85547c0ad834168110cbc6d P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } }
I20240501 07:58:58.833871 7839 raft_consensus.cc:374] T e651f523d85547c0ad834168110cbc6d P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:58.833948 7839 raft_consensus.cc:729] T e651f523d85547c0ad834168110cbc6d P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8315d22c56f7439a9e1fc82f153f018c, State: Initialized, Role: FOLLOWER
I20240501 07:58:58.834007 7838 tablet_bootstrap.cc:492] T c3a9036ccfee45498bca9681518d3699 P 3b164cbbc00f4e8eb3254ddd349a3ff9: No bootstrap required, opened a new log
I20240501 07:58:58.834105 7838 ts_tablet_manager.cc:1367] T c3a9036ccfee45498bca9681518d3699 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20240501 07:58:58.834332 7842 tablet_bootstrap.cc:492] T c3a9036ccfee45498bca9681518d3699 P 1f9cb72be98941b5a8e504cbe34132b9: No bootstrap required, opened a new log
I20240501 07:58:58.834252 7839 consensus_queue.cc:260] T e651f523d85547c0ad834168110cbc6d P 8315d22c56f7439a9e1fc82f153f018c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } }
I20240501 07:58:58.834419 7842 ts_tablet_manager.cc:1367] T c3a9036ccfee45498bca9681518d3699 P 1f9cb72be98941b5a8e504cbe34132b9: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.002s
I20240501 07:58:58.834575 7838 raft_consensus.cc:348] T c3a9036ccfee45498bca9681518d3699 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:58:58.834674 7838 raft_consensus.cc:374] T c3a9036ccfee45498bca9681518d3699 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:58.834673 7839 ts_tablet_manager.cc:1398] T e651f523d85547c0ad834168110cbc6d P 8315d22c56f7439a9e1fc82f153f018c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:58.834707 7838 raft_consensus.cc:729] T c3a9036ccfee45498bca9681518d3699 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b164cbbc00f4e8eb3254ddd349a3ff9, State: Initialized, Role: FOLLOWER
I20240501 07:58:58.834837 7839 tablet_bootstrap.cc:492] T c3a9036ccfee45498bca9681518d3699 P 8315d22c56f7439a9e1fc82f153f018c: Bootstrap starting.
I20240501 07:58:58.834820 7838 consensus_queue.cc:260] T c3a9036ccfee45498bca9681518d3699 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:58:58.834903 7842 raft_consensus.cc:348] T c3a9036ccfee45498bca9681518d3699 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:58:58.834993 7842 raft_consensus.cc:374] T c3a9036ccfee45498bca9681518d3699 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:58.835022 7842 raft_consensus.cc:729] T c3a9036ccfee45498bca9681518d3699 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f9cb72be98941b5a8e504cbe34132b9, State: Initialized, Role: FOLLOWER
I20240501 07:58:58.835103 7838 ts_tablet_manager.cc:1398] T c3a9036ccfee45498bca9681518d3699 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:58:58.835129 7842 consensus_queue.cc:260] T c3a9036ccfee45498bca9681518d3699 P 1f9cb72be98941b5a8e504cbe34132b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:58:58.835419 7842 ts_tablet_manager.cc:1398] T c3a9036ccfee45498bca9681518d3699 P 1f9cb72be98941b5a8e504cbe34132b9: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 07:58:58.835916 7839 tablet_bootstrap.cc:654] T c3a9036ccfee45498bca9681518d3699 P 8315d22c56f7439a9e1fc82f153f018c: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:58.841461 7839 tablet_bootstrap.cc:492] T c3a9036ccfee45498bca9681518d3699 P 8315d22c56f7439a9e1fc82f153f018c: No bootstrap required, opened a new log
I20240501 07:58:58.841567 7839 ts_tablet_manager.cc:1367] T c3a9036ccfee45498bca9681518d3699 P 8315d22c56f7439a9e1fc82f153f018c: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240501 07:58:58.842056 7839 raft_consensus.cc:348] T c3a9036ccfee45498bca9681518d3699 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:58:58.842171 7839 raft_consensus.cc:374] T c3a9036ccfee45498bca9681518d3699 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:58.842216 7839 raft_consensus.cc:729] T c3a9036ccfee45498bca9681518d3699 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8315d22c56f7439a9e1fc82f153f018c, State: Initialized, Role: FOLLOWER
I20240501 07:58:58.842330 7839 consensus_queue.cc:260] T c3a9036ccfee45498bca9681518d3699 P 8315d22c56f7439a9e1fc82f153f018c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:58:58.842607 7839 ts_tablet_manager.cc:1398] T c3a9036ccfee45498bca9681518d3699 P 8315d22c56f7439a9e1fc82f153f018c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:58.891840 7833 raft_consensus.cc:482] T c3a9036ccfee45498bca9681518d3699 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:58.891985 7833 raft_consensus.cc:504] T c3a9036ccfee45498bca9681518d3699 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:58:58.892397 7833 leader_election.cc:288] T c3a9036ccfee45498bca9681518d3699 P 1f9cb72be98941b5a8e504cbe34132b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131:34471), 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130:32941)
I20240501 07:58:58.892813 6576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3a9036ccfee45498bca9681518d3699" candidate_uuid: "1f9cb72be98941b5a8e504cbe34132b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" is_pre_election: true
I20240501 07:58:58.892832 6445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3a9036ccfee45498bca9681518d3699" candidate_uuid: "1f9cb72be98941b5a8e504cbe34132b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8315d22c56f7439a9e1fc82f153f018c" is_pre_election: true
I20240501 07:58:58.892946 6576 raft_consensus.cc:2454] T c3a9036ccfee45498bca9681518d3699 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f9cb72be98941b5a8e504cbe34132b9 in term 0.
I20240501 07:58:58.892989 6445 raft_consensus.cc:2454] T c3a9036ccfee45498bca9681518d3699 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f9cb72be98941b5a8e504cbe34132b9 in term 0.
I20240501 07:58:58.893283 6644 leader_election.cc:302] T c3a9036ccfee45498bca9681518d3699 P 1f9cb72be98941b5a8e504cbe34132b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f9cb72be98941b5a8e504cbe34132b9, 3b164cbbc00f4e8eb3254ddd349a3ff9; no voters:
I20240501 07:58:58.893460 7833 raft_consensus.cc:2781] T c3a9036ccfee45498bca9681518d3699 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:58.893520 7833 raft_consensus.cc:482] T c3a9036ccfee45498bca9681518d3699 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:58.893548 7833 raft_consensus.cc:3037] T c3a9036ccfee45498bca9681518d3699 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:58.894414 7833 raft_consensus.cc:504] T c3a9036ccfee45498bca9681518d3699 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:58:58.894785 7833 leader_election.cc:288] T c3a9036ccfee45498bca9681518d3699 P 1f9cb72be98941b5a8e504cbe34132b9 [CANDIDATE]: Term 1 election: Requested vote from peers 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131:34471), 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130:32941)
I20240501 07:58:58.895153 6576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3a9036ccfee45498bca9681518d3699" candidate_uuid: "1f9cb72be98941b5a8e504cbe34132b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9"
I20240501 07:58:58.895239 6445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3a9036ccfee45498bca9681518d3699" candidate_uuid: "1f9cb72be98941b5a8e504cbe34132b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8315d22c56f7439a9e1fc82f153f018c"
I20240501 07:58:58.895287 6576 raft_consensus.cc:3037] T c3a9036ccfee45498bca9681518d3699 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:58.895361 6445 raft_consensus.cc:3037] T c3a9036ccfee45498bca9681518d3699 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:58.896181 6576 raft_consensus.cc:2454] T c3a9036ccfee45498bca9681518d3699 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f9cb72be98941b5a8e504cbe34132b9 in term 1.
I20240501 07:58:58.896205 6445 raft_consensus.cc:2454] T c3a9036ccfee45498bca9681518d3699 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f9cb72be98941b5a8e504cbe34132b9 in term 1.
I20240501 07:58:58.896473 6644 leader_election.cc:302] T c3a9036ccfee45498bca9681518d3699 P 1f9cb72be98941b5a8e504cbe34132b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f9cb72be98941b5a8e504cbe34132b9, 3b164cbbc00f4e8eb3254ddd349a3ff9; no voters:
I20240501 07:58:58.896649 7833 raft_consensus.cc:2781] T c3a9036ccfee45498bca9681518d3699 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:58.896749 7833 raft_consensus.cc:686] T c3a9036ccfee45498bca9681518d3699 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 LEADER]: Becoming Leader. State: Replica: 1f9cb72be98941b5a8e504cbe34132b9, State: Running, Role: LEADER
I20240501 07:58:58.896919 7833 consensus_queue.cc:237] T c3a9036ccfee45498bca9681518d3699 P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:58:58.898960 6169 catalog_manager.cc:5484] T c3a9036ccfee45498bca9681518d3699 P 1f9cb72be98941b5a8e504cbe34132b9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132). New cstate: current_term: 1 leader_uuid: "1f9cb72be98941b5a8e504cbe34132b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } health_report { overall_health: HEALTHY } } }
I20240501 07:58:58.910537 7833 raft_consensus.cc:482] T 8454f24b0c28451b8ee51be39ca3e07f P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:58.910646 7833 raft_consensus.cc:504] T 8454f24b0c28451b8ee51be39ca3e07f P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } }
I20240501 07:58:58.911015 7833 leader_election.cc:288] T 8454f24b0c28451b8ee51be39ca3e07f P 1f9cb72be98941b5a8e504cbe34132b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131:34471), 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129:46875)
I20240501 07:58:58.911345 6576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8454f24b0c28451b8ee51be39ca3e07f" candidate_uuid: "1f9cb72be98941b5a8e504cbe34132b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" is_pre_election: true
I20240501 07:58:58.911376 6315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8454f24b0c28451b8ee51be39ca3e07f" candidate_uuid: "1f9cb72be98941b5a8e504cbe34132b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" is_pre_election: true
I20240501 07:58:58.911494 6576 raft_consensus.cc:2454] T 8454f24b0c28451b8ee51be39ca3e07f P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f9cb72be98941b5a8e504cbe34132b9 in term 0.
I20240501 07:58:58.911494 6315 raft_consensus.cc:2454] T 8454f24b0c28451b8ee51be39ca3e07f P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f9cb72be98941b5a8e504cbe34132b9 in term 0.
I20240501 07:58:58.911751 6641 leader_election.cc:302] T 8454f24b0c28451b8ee51be39ca3e07f P 1f9cb72be98941b5a8e504cbe34132b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f9cb72be98941b5a8e504cbe34132b9, 8b8b3ad3aa8f4053b2167dd550f2086f; no voters:
I20240501 07:58:58.911906 7833 raft_consensus.cc:2781] T 8454f24b0c28451b8ee51be39ca3e07f P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:58.911967 7833 raft_consensus.cc:482] T 8454f24b0c28451b8ee51be39ca3e07f P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:58.912009 7833 raft_consensus.cc:3037] T 8454f24b0c28451b8ee51be39ca3e07f P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:58.912796 7833 raft_consensus.cc:504] T 8454f24b0c28451b8ee51be39ca3e07f P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } }
I20240501 07:58:58.913106 7833 leader_election.cc:288] T 8454f24b0c28451b8ee51be39ca3e07f P 1f9cb72be98941b5a8e504cbe34132b9 [CANDIDATE]: Term 1 election: Requested vote from peers 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131:34471), 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129:46875)
I20240501 07:58:58.913388 6576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8454f24b0c28451b8ee51be39ca3e07f" candidate_uuid: "1f9cb72be98941b5a8e504cbe34132b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9"
I20240501 07:58:58.913416 6315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8454f24b0c28451b8ee51be39ca3e07f" candidate_uuid: "1f9cb72be98941b5a8e504cbe34132b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f"
I20240501 07:58:58.913517 6315 raft_consensus.cc:3037] T 8454f24b0c28451b8ee51be39ca3e07f P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:58.913512 6576 raft_consensus.cc:3037] T 8454f24b0c28451b8ee51be39ca3e07f P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:58.914287 6576 raft_consensus.cc:2454] T 8454f24b0c28451b8ee51be39ca3e07f P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f9cb72be98941b5a8e504cbe34132b9 in term 1.
I20240501 07:58:58.914378 6315 raft_consensus.cc:2454] T 8454f24b0c28451b8ee51be39ca3e07f P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f9cb72be98941b5a8e504cbe34132b9 in term 1.
I20240501 07:58:58.914541 6644 leader_election.cc:302] T 8454f24b0c28451b8ee51be39ca3e07f P 1f9cb72be98941b5a8e504cbe34132b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f9cb72be98941b5a8e504cbe34132b9, 3b164cbbc00f4e8eb3254ddd349a3ff9; no voters:
I20240501 07:58:58.914695 7833 raft_consensus.cc:2781] T 8454f24b0c28451b8ee51be39ca3e07f P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:58.914778 7833 raft_consensus.cc:686] T 8454f24b0c28451b8ee51be39ca3e07f P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 LEADER]: Becoming Leader. State: Replica: 1f9cb72be98941b5a8e504cbe34132b9, State: Running, Role: LEADER
I20240501 07:58:58.914913 7833 consensus_queue.cc:237] T 8454f24b0c28451b8ee51be39ca3e07f P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } }
I20240501 07:58:58.916579 6169 catalog_manager.cc:5484] T 8454f24b0c28451b8ee51be39ca3e07f P 1f9cb72be98941b5a8e504cbe34132b9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132). New cstate: current_term: 1 leader_uuid: "1f9cb72be98941b5a8e504cbe34132b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } health_report { overall_health: UNKNOWN } } }
I20240501 07:58:59.012593 7321 raft_consensus.cc:482] T e651f523d85547c0ad834168110cbc6d P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:59.012735 7321 raft_consensus.cc:504] T e651f523d85547c0ad834168110cbc6d P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } }
I20240501 07:58:59.013178 7321 leader_election.cc:288] T e651f523d85547c0ad834168110cbc6d P 3b164cbbc00f4e8eb3254ddd349a3ff9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129:46875), 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130:32941)
I20240501 07:58:59.013566 6445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e651f523d85547c0ad834168110cbc6d" candidate_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8315d22c56f7439a9e1fc82f153f018c" is_pre_election: true
I20240501 07:58:59.013590 6315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e651f523d85547c0ad834168110cbc6d" candidate_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" is_pre_election: true
I20240501 07:58:59.013720 6315 raft_consensus.cc:2454] T e651f523d85547c0ad834168110cbc6d P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b164cbbc00f4e8eb3254ddd349a3ff9 in term 0.
I20240501 07:58:59.013727 6445 raft_consensus.cc:2454] T e651f523d85547c0ad834168110cbc6d P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b164cbbc00f4e8eb3254ddd349a3ff9 in term 0.
I20240501 07:58:59.014020 6510 leader_election.cc:302] T e651f523d85547c0ad834168110cbc6d P 3b164cbbc00f4e8eb3254ddd349a3ff9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b164cbbc00f4e8eb3254ddd349a3ff9, 8b8b3ad3aa8f4053b2167dd550f2086f; no voters:
I20240501 07:58:59.014219 7321 raft_consensus.cc:2781] T e651f523d85547c0ad834168110cbc6d P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:59.014292 7321 raft_consensus.cc:482] T e651f523d85547c0ad834168110cbc6d P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:59.014346 7321 raft_consensus.cc:3037] T e651f523d85547c0ad834168110cbc6d P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:59.015337 7321 raft_consensus.cc:504] T e651f523d85547c0ad834168110cbc6d P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } }
I20240501 07:58:59.015664 7321 leader_election.cc:288] T e651f523d85547c0ad834168110cbc6d P 3b164cbbc00f4e8eb3254ddd349a3ff9 [CANDIDATE]: Term 1 election: Requested vote from peers 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129:46875), 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130:32941)
I20240501 07:58:59.016009 6445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e651f523d85547c0ad834168110cbc6d" candidate_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8315d22c56f7439a9e1fc82f153f018c"
I20240501 07:58:59.016028 6315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e651f523d85547c0ad834168110cbc6d" candidate_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f"
I20240501 07:58:59.016127 6445 raft_consensus.cc:3037] T e651f523d85547c0ad834168110cbc6d P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:59.016127 6315 raft_consensus.cc:3037] T e651f523d85547c0ad834168110cbc6d P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:59.017084 6315 raft_consensus.cc:2454] T e651f523d85547c0ad834168110cbc6d P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b164cbbc00f4e8eb3254ddd349a3ff9 in term 1.
I20240501 07:58:59.017086 6445 raft_consensus.cc:2454] T e651f523d85547c0ad834168110cbc6d P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b164cbbc00f4e8eb3254ddd349a3ff9 in term 1.
I20240501 07:58:59.017390 6512 leader_election.cc:302] T e651f523d85547c0ad834168110cbc6d P 3b164cbbc00f4e8eb3254ddd349a3ff9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b164cbbc00f4e8eb3254ddd349a3ff9, 8315d22c56f7439a9e1fc82f153f018c; no voters:
I20240501 07:58:59.017563 7321 raft_consensus.cc:2781] T e651f523d85547c0ad834168110cbc6d P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:59.017796 7321 raft_consensus.cc:686] T e651f523d85547c0ad834168110cbc6d P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 LEADER]: Becoming Leader. State: Replica: 3b164cbbc00f4e8eb3254ddd349a3ff9, State: Running, Role: LEADER
I20240501 07:58:59.017951 7321 consensus_queue.cc:237] T e651f523d85547c0ad834168110cbc6d P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } }
I20240501 07:58:59.019642 6169 catalog_manager.cc:5484] T e651f523d85547c0ad834168110cbc6d P 3b164cbbc00f4e8eb3254ddd349a3ff9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } health_report { overall_health: HEALTHY } } }
W20240501 07:58:59.168933 7855 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:58:59.169268 7855 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:58:59.172328 7855 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_7'
I20240501 07:58:59.264680 6445 raft_consensus.cc:1261] T c3a9036ccfee45498bca9681518d3699 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Refusing update from remote peer 1f9cb72be98941b5a8e504cbe34132b9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:59.265072 6576 raft_consensus.cc:1261] T 8454f24b0c28451b8ee51be39ca3e07f P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Refusing update from remote peer 1f9cb72be98941b5a8e504cbe34132b9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:59.265208 7828 consensus_queue.cc:1035] T c3a9036ccfee45498bca9681518d3699 P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:59.265465 6315 raft_consensus.cc:1261] T 8454f24b0c28451b8ee51be39ca3e07f P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Refusing update from remote peer 1f9cb72be98941b5a8e504cbe34132b9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:59.266105 6577 raft_consensus.cc:1261] T c3a9036ccfee45498bca9681518d3699 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Refusing update from remote peer 1f9cb72be98941b5a8e504cbe34132b9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:59.267061 7828 consensus_queue.cc:1035] T 8454f24b0c28451b8ee51be39ca3e07f P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:59.269661 7467 consensus_queue.cc:1035] T 8454f24b0c28451b8ee51be39ca3e07f P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:59.269950 7467 consensus_queue.cc:1035] T c3a9036ccfee45498bca9681518d3699 P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:59.272099 6313 raft_consensus.cc:1261] T e651f523d85547c0ad834168110cbc6d P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Refusing update from remote peer 3b164cbbc00f4e8eb3254ddd349a3ff9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:59.272514 7321 consensus_queue.cc:1035] T e651f523d85547c0ad834168110cbc6d P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:59.273726 6446 raft_consensus.cc:1261] T e651f523d85547c0ad834168110cbc6d P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Refusing update from remote peer 3b164cbbc00f4e8eb3254ddd349a3ff9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:59.274674 7853 consensus_queue.cc:1035] T e651f523d85547c0ad834168110cbc6d P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:59.281358 7888 mvcc.cc:205] Tried to move back new op lower bound from 7022798189622370304 to 7022798188123238400. Current Snapshot: MvccSnapshot[applied={T|T < 7022798189622370304}]
INSERT report
rows total: 2000
time total: 119.987 ms
time per row: 0.0599935 ms
I20240501 07:58:59.336457 6167 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55950:
name: "rebalance_test_table_8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 07:58:59.342443 6557 tablet_service.cc:1467] Processing CreateTablet for tablet 12508ef4d9c942939ce08c14344754b4 (DEFAULT_TABLE table=rebalance_test_table_8 [id=85a512f46553493eaf7ce4a1eecac55f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:59.342505 6294 tablet_service.cc:1467] Processing CreateTablet for tablet 12508ef4d9c942939ce08c14344754b4 (DEFAULT_TABLE table=rebalance_test_table_8 [id=85a512f46553493eaf7ce4a1eecac55f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:59.342797 6557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12508ef4d9c942939ce08c14344754b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:59.342799 6294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12508ef4d9c942939ce08c14344754b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:59.343677 6295 tablet_service.cc:1467] Processing CreateTablet for tablet 0cf117c67b524add96c9e2d887dc9e4d (DEFAULT_TABLE table=rebalance_test_table_8 [id=85a512f46553493eaf7ce4a1eecac55f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:59.343777 6426 tablet_service.cc:1467] Processing CreateTablet for tablet 0cf117c67b524add96c9e2d887dc9e4d (DEFAULT_TABLE table=rebalance_test_table_8 [id=85a512f46553493eaf7ce4a1eecac55f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:59.344062 6426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0cf117c67b524add96c9e2d887dc9e4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:59.344039 6687 tablet_service.cc:1467] Processing CreateTablet for tablet 0cf117c67b524add96c9e2d887dc9e4d (DEFAULT_TABLE table=rebalance_test_table_8 [id=85a512f46553493eaf7ce4a1eecac55f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:59.344049 6293 tablet_service.cc:1467] Processing CreateTablet for tablet 7d51b1940b594c79a7d13d7026b76da8 (DEFAULT_TABLE table=rebalance_test_table_8 [id=85a512f46553493eaf7ce4a1eecac55f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:59.344272 6687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0cf117c67b524add96c9e2d887dc9e4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:59.345089 6293 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d51b1940b594c79a7d13d7026b76da8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:59.345407 6295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0cf117c67b524add96c9e2d887dc9e4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:59.346022 7900 tablet_bootstrap.cc:492] T 12508ef4d9c942939ce08c14344754b4 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Bootstrap starting.
I20240501 07:58:59.346565 6556 tablet_service.cc:1467] Processing CreateTablet for tablet 7d51b1940b594c79a7d13d7026b76da8 (DEFAULT_TABLE table=rebalance_test_table_8 [id=85a512f46553493eaf7ce4a1eecac55f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:59.346791 6556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d51b1940b594c79a7d13d7026b76da8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:59.346971 7900 tablet_bootstrap.cc:654] T 12508ef4d9c942939ce08c14344754b4 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:59.347496 7903 tablet_bootstrap.cc:492] T 0cf117c67b524add96c9e2d887dc9e4d P 8b8b3ad3aa8f4053b2167dd550f2086f: Bootstrap starting.
I20240501 07:58:59.348469 7903 tablet_bootstrap.cc:654] T 0cf117c67b524add96c9e2d887dc9e4d P 8b8b3ad3aa8f4053b2167dd550f2086f: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:59.348788 6688 tablet_service.cc:1467] Processing CreateTablet for tablet 7d51b1940b594c79a7d13d7026b76da8 (DEFAULT_TABLE table=rebalance_test_table_8 [id=85a512f46553493eaf7ce4a1eecac55f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:59.343780 6424 tablet_service.cc:1467] Processing CreateTablet for tablet 12508ef4d9c942939ce08c14344754b4 (DEFAULT_TABLE table=rebalance_test_table_8 [id=85a512f46553493eaf7ce4a1eecac55f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:59.349351 6424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12508ef4d9c942939ce08c14344754b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:59.350553 6688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d51b1940b594c79a7d13d7026b76da8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:59.350800 7900 tablet_bootstrap.cc:492] T 12508ef4d9c942939ce08c14344754b4 P 3b164cbbc00f4e8eb3254ddd349a3ff9: No bootstrap required, opened a new log
I20240501 07:58:59.350888 7900 ts_tablet_manager.cc:1367] T 12508ef4d9c942939ce08c14344754b4 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 07:58:59.351302 7900 raft_consensus.cc:348] T 12508ef4d9c942939ce08c14344754b4 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } }
I20240501 07:58:59.351439 7900 raft_consensus.cc:374] T 12508ef4d9c942939ce08c14344754b4 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:59.351500 7900 raft_consensus.cc:729] T 12508ef4d9c942939ce08c14344754b4 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b164cbbc00f4e8eb3254ddd349a3ff9, State: Initialized, Role: FOLLOWER
I20240501 07:58:59.351642 7900 consensus_queue.cc:260] T 12508ef4d9c942939ce08c14344754b4 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } }
I20240501 07:58:59.351934 7900 ts_tablet_manager.cc:1398] T 12508ef4d9c942939ce08c14344754b4 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:59.352095 7900 tablet_bootstrap.cc:492] T 7d51b1940b594c79a7d13d7026b76da8 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Bootstrap starting.
I20240501 07:58:59.353014 7902 tablet_bootstrap.cc:492] T 0cf117c67b524add96c9e2d887dc9e4d P 1f9cb72be98941b5a8e504cbe34132b9: Bootstrap starting.
I20240501 07:58:59.353156 7901 tablet_bootstrap.cc:492] T 0cf117c67b524add96c9e2d887dc9e4d P 8315d22c56f7439a9e1fc82f153f018c: Bootstrap starting.
I20240501 07:58:59.353891 7902 tablet_bootstrap.cc:654] T 0cf117c67b524add96c9e2d887dc9e4d P 1f9cb72be98941b5a8e504cbe34132b9: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:59.354259 7900 tablet_bootstrap.cc:654] T 7d51b1940b594c79a7d13d7026b76da8 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:59.354310 7901 tablet_bootstrap.cc:654] T 0cf117c67b524add96c9e2d887dc9e4d P 8315d22c56f7439a9e1fc82f153f018c: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:59.355693 7900 tablet_bootstrap.cc:492] T 7d51b1940b594c79a7d13d7026b76da8 P 3b164cbbc00f4e8eb3254ddd349a3ff9: No bootstrap required, opened a new log
I20240501 07:58:59.355825 7900 ts_tablet_manager.cc:1367] T 7d51b1940b594c79a7d13d7026b76da8 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240501 07:58:59.355845 7903 tablet_bootstrap.cc:492] T 0cf117c67b524add96c9e2d887dc9e4d P 8b8b3ad3aa8f4053b2167dd550f2086f: No bootstrap required, opened a new log
I20240501 07:58:59.355934 7903 ts_tablet_manager.cc:1367] T 0cf117c67b524add96c9e2d887dc9e4d P 8b8b3ad3aa8f4053b2167dd550f2086f: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240501 07:58:59.356083 7902 tablet_bootstrap.cc:492] T 0cf117c67b524add96c9e2d887dc9e4d P 1f9cb72be98941b5a8e504cbe34132b9: No bootstrap required, opened a new log
I20240501 07:58:59.356158 7902 ts_tablet_manager.cc:1367] T 0cf117c67b524add96c9e2d887dc9e4d P 1f9cb72be98941b5a8e504cbe34132b9: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 07:58:59.356320 7900 raft_consensus.cc:348] T 7d51b1940b594c79a7d13d7026b76da8 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } }
I20240501 07:58:59.356447 7900 raft_consensus.cc:374] T 7d51b1940b594c79a7d13d7026b76da8 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:59.356482 7900 raft_consensus.cc:729] T 7d51b1940b594c79a7d13d7026b76da8 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b164cbbc00f4e8eb3254ddd349a3ff9, State: Initialized, Role: FOLLOWER
I20240501 07:58:59.356588 7902 raft_consensus.cc:348] T 0cf117c67b524add96c9e2d887dc9e4d P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } }
I20240501 07:58:59.356678 7902 raft_consensus.cc:374] T 0cf117c67b524add96c9e2d887dc9e4d P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:59.356709 7902 raft_consensus.cc:729] T 0cf117c67b524add96c9e2d887dc9e4d P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f9cb72be98941b5a8e504cbe34132b9, State: Initialized, Role: FOLLOWER
I20240501 07:58:59.356815 7902 consensus_queue.cc:260] T 0cf117c67b524add96c9e2d887dc9e4d P 1f9cb72be98941b5a8e504cbe34132b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } }
I20240501 07:58:59.357209 7902 ts_tablet_manager.cc:1398] T 0cf117c67b524add96c9e2d887dc9e4d P 1f9cb72be98941b5a8e504cbe34132b9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:59.357421 7902 tablet_bootstrap.cc:492] T 7d51b1940b594c79a7d13d7026b76da8 P 1f9cb72be98941b5a8e504cbe34132b9: Bootstrap starting.
I20240501 07:58:59.356629 7900 consensus_queue.cc:260] T 7d51b1940b594c79a7d13d7026b76da8 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } }
I20240501 07:58:59.357775 7900 ts_tablet_manager.cc:1398] T 7d51b1940b594c79a7d13d7026b76da8 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 07:58:59.358250 7902 tablet_bootstrap.cc:654] T 7d51b1940b594c79a7d13d7026b76da8 P 1f9cb72be98941b5a8e504cbe34132b9: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:59.356410 7903 raft_consensus.cc:348] T 0cf117c67b524add96c9e2d887dc9e4d P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } }
I20240501 07:58:59.359092 7903 raft_consensus.cc:374] T 0cf117c67b524add96c9e2d887dc9e4d P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:59.359144 7903 raft_consensus.cc:729] T 0cf117c67b524add96c9e2d887dc9e4d P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b8b3ad3aa8f4053b2167dd550f2086f, State: Initialized, Role: FOLLOWER
I20240501 07:58:59.359208 7902 tablet_bootstrap.cc:492] T 7d51b1940b594c79a7d13d7026b76da8 P 1f9cb72be98941b5a8e504cbe34132b9: No bootstrap required, opened a new log
I20240501 07:58:59.359274 7902 ts_tablet_manager.cc:1367] T 7d51b1940b594c79a7d13d7026b76da8 P 1f9cb72be98941b5a8e504cbe34132b9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:58:59.359294 7903 consensus_queue.cc:260] T 0cf117c67b524add96c9e2d887dc9e4d P 8b8b3ad3aa8f4053b2167dd550f2086f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } }
I20240501 07:58:59.359766 7903 ts_tablet_manager.cc:1398] T 0cf117c67b524add96c9e2d887dc9e4d P 8b8b3ad3aa8f4053b2167dd550f2086f: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240501 07:58:59.359970 7903 tablet_bootstrap.cc:492] T 7d51b1940b594c79a7d13d7026b76da8 P 8b8b3ad3aa8f4053b2167dd550f2086f: Bootstrap starting.
I20240501 07:58:59.360153 7902 raft_consensus.cc:348] T 7d51b1940b594c79a7d13d7026b76da8 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } }
I20240501 07:58:59.360308 7902 raft_consensus.cc:374] T 7d51b1940b594c79a7d13d7026b76da8 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:59.360359 7902 raft_consensus.cc:729] T 7d51b1940b594c79a7d13d7026b76da8 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f9cb72be98941b5a8e504cbe34132b9, State: Initialized, Role: FOLLOWER
I20240501 07:58:59.360504 7902 consensus_queue.cc:260] T 7d51b1940b594c79a7d13d7026b76da8 P 1f9cb72be98941b5a8e504cbe34132b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } }
I20240501 07:58:59.360836 7902 ts_tablet_manager.cc:1398] T 7d51b1940b594c79a7d13d7026b76da8 P 1f9cb72be98941b5a8e504cbe34132b9: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:58:59.361472 7901 tablet_bootstrap.cc:492] T 0cf117c67b524add96c9e2d887dc9e4d P 8315d22c56f7439a9e1fc82f153f018c: No bootstrap required, opened a new log
I20240501 07:58:59.361572 7901 ts_tablet_manager.cc:1367] T 0cf117c67b524add96c9e2d887dc9e4d P 8315d22c56f7439a9e1fc82f153f018c: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240501 07:58:59.361999 7901 raft_consensus.cc:348] T 0cf117c67b524add96c9e2d887dc9e4d P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } }
I20240501 07:58:59.362099 7901 raft_consensus.cc:374] T 0cf117c67b524add96c9e2d887dc9e4d P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:59.362128 7901 raft_consensus.cc:729] T 0cf117c67b524add96c9e2d887dc9e4d P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8315d22c56f7439a9e1fc82f153f018c, State: Initialized, Role: FOLLOWER
I20240501 07:58:59.362252 7901 consensus_queue.cc:260] T 0cf117c67b524add96c9e2d887dc9e4d P 8315d22c56f7439a9e1fc82f153f018c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } }
I20240501 07:58:59.362534 7901 ts_tablet_manager.cc:1398] T 0cf117c67b524add96c9e2d887dc9e4d P 8315d22c56f7439a9e1fc82f153f018c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:58:59.362716 7901 tablet_bootstrap.cc:492] T 12508ef4d9c942939ce08c14344754b4 P 8315d22c56f7439a9e1fc82f153f018c: Bootstrap starting.
I20240501 07:58:59.362335 7903 tablet_bootstrap.cc:654] T 7d51b1940b594c79a7d13d7026b76da8 P 8b8b3ad3aa8f4053b2167dd550f2086f: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:59.363755 7901 tablet_bootstrap.cc:654] T 12508ef4d9c942939ce08c14344754b4 P 8315d22c56f7439a9e1fc82f153f018c: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:59.364941 7901 tablet_bootstrap.cc:492] T 12508ef4d9c942939ce08c14344754b4 P 8315d22c56f7439a9e1fc82f153f018c: No bootstrap required, opened a new log
I20240501 07:58:59.365078 7901 ts_tablet_manager.cc:1367] T 12508ef4d9c942939ce08c14344754b4 P 8315d22c56f7439a9e1fc82f153f018c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 07:58:59.365304 7903 tablet_bootstrap.cc:492] T 7d51b1940b594c79a7d13d7026b76da8 P 8b8b3ad3aa8f4053b2167dd550f2086f: No bootstrap required, opened a new log
I20240501 07:58:59.365442 7903 ts_tablet_manager.cc:1367] T 7d51b1940b594c79a7d13d7026b76da8 P 8b8b3ad3aa8f4053b2167dd550f2086f: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.002s
I20240501 07:58:59.365553 7901 raft_consensus.cc:348] T 12508ef4d9c942939ce08c14344754b4 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } }
I20240501 07:58:59.365666 7901 raft_consensus.cc:374] T 12508ef4d9c942939ce08c14344754b4 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:59.365705 7901 raft_consensus.cc:729] T 12508ef4d9c942939ce08c14344754b4 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8315d22c56f7439a9e1fc82f153f018c, State: Initialized, Role: FOLLOWER
I20240501 07:58:59.365859 7901 consensus_queue.cc:260] T 12508ef4d9c942939ce08c14344754b4 P 8315d22c56f7439a9e1fc82f153f018c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } }
I20240501 07:58:59.366091 7903 raft_consensus.cc:348] T 7d51b1940b594c79a7d13d7026b76da8 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } }
I20240501 07:58:59.366150 7901 ts_tablet_manager.cc:1398] T 12508ef4d9c942939ce08c14344754b4 P 8315d22c56f7439a9e1fc82f153f018c: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 07:58:59.366182 7903 raft_consensus.cc:374] T 7d51b1940b594c79a7d13d7026b76da8 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:59.366218 7903 raft_consensus.cc:729] T 7d51b1940b594c79a7d13d7026b76da8 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b8b3ad3aa8f4053b2167dd550f2086f, State: Initialized, Role: FOLLOWER
I20240501 07:58:59.366365 7903 consensus_queue.cc:260] T 7d51b1940b594c79a7d13d7026b76da8 P 8b8b3ad3aa8f4053b2167dd550f2086f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } }
I20240501 07:58:59.366645 7903 ts_tablet_manager.cc:1398] T 7d51b1940b594c79a7d13d7026b76da8 P 8b8b3ad3aa8f4053b2167dd550f2086f: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 07:58:59.366858 7903 tablet_bootstrap.cc:492] T 12508ef4d9c942939ce08c14344754b4 P 8b8b3ad3aa8f4053b2167dd550f2086f: Bootstrap starting.
I20240501 07:58:59.367841 7903 tablet_bootstrap.cc:654] T 12508ef4d9c942939ce08c14344754b4 P 8b8b3ad3aa8f4053b2167dd550f2086f: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:59.368924 7903 tablet_bootstrap.cc:492] T 12508ef4d9c942939ce08c14344754b4 P 8b8b3ad3aa8f4053b2167dd550f2086f: No bootstrap required, opened a new log
I20240501 07:58:59.369028 7903 ts_tablet_manager.cc:1367] T 12508ef4d9c942939ce08c14344754b4 P 8b8b3ad3aa8f4053b2167dd550f2086f: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240501 07:58:59.369365 7903 raft_consensus.cc:348] T 12508ef4d9c942939ce08c14344754b4 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } }
I20240501 07:58:59.369441 7903 raft_consensus.cc:374] T 12508ef4d9c942939ce08c14344754b4 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:59.369462 7903 raft_consensus.cc:729] T 12508ef4d9c942939ce08c14344754b4 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b8b3ad3aa8f4053b2167dd550f2086f, State: Initialized, Role: FOLLOWER
I20240501 07:58:59.369565 7903 consensus_queue.cc:260] T 12508ef4d9c942939ce08c14344754b4 P 8b8b3ad3aa8f4053b2167dd550f2086f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } }
I20240501 07:58:59.369838 7903 ts_tablet_manager.cc:1398] T 12508ef4d9c942939ce08c14344754b4 P 8b8b3ad3aa8f4053b2167dd550f2086f: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 07:58:59.436995 7467 raft_consensus.cc:482] T 7d51b1940b594c79a7d13d7026b76da8 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:59.437145 7467 raft_consensus.cc:504] T 7d51b1940b594c79a7d13d7026b76da8 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } }
I20240501 07:58:59.437592 7467 leader_election.cc:288] T 7d51b1940b594c79a7d13d7026b76da8 P 1f9cb72be98941b5a8e504cbe34132b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129:46875), 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131:34471)
I20240501 07:58:59.438072 6576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d51b1940b594c79a7d13d7026b76da8" candidate_uuid: "1f9cb72be98941b5a8e504cbe34132b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" is_pre_election: true
I20240501 07:58:59.438074 6313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d51b1940b594c79a7d13d7026b76da8" candidate_uuid: "1f9cb72be98941b5a8e504cbe34132b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" is_pre_election: true
I20240501 07:58:59.438212 6576 raft_consensus.cc:2454] T 7d51b1940b594c79a7d13d7026b76da8 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f9cb72be98941b5a8e504cbe34132b9 in term 0.
I20240501 07:58:59.438225 6313 raft_consensus.cc:2454] T 7d51b1940b594c79a7d13d7026b76da8 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f9cb72be98941b5a8e504cbe34132b9 in term 0.
I20240501 07:58:59.438525 6644 leader_election.cc:302] T 7d51b1940b594c79a7d13d7026b76da8 P 1f9cb72be98941b5a8e504cbe34132b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f9cb72be98941b5a8e504cbe34132b9, 3b164cbbc00f4e8eb3254ddd349a3ff9; no voters:
I20240501 07:58:59.438668 7467 raft_consensus.cc:2781] T 7d51b1940b594c79a7d13d7026b76da8 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:59.438725 7467 raft_consensus.cc:482] T 7d51b1940b594c79a7d13d7026b76da8 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:59.438776 7467 raft_consensus.cc:3037] T 7d51b1940b594c79a7d13d7026b76da8 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:59.439639 7467 raft_consensus.cc:504] T 7d51b1940b594c79a7d13d7026b76da8 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } }
I20240501 07:58:59.439947 7467 leader_election.cc:288] T 7d51b1940b594c79a7d13d7026b76da8 P 1f9cb72be98941b5a8e504cbe34132b9 [CANDIDATE]: Term 1 election: Requested vote from peers 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129:46875), 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131:34471)
I20240501 07:58:59.440322 6576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d51b1940b594c79a7d13d7026b76da8" candidate_uuid: "1f9cb72be98941b5a8e504cbe34132b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9"
I20240501 07:58:59.440315 6313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d51b1940b594c79a7d13d7026b76da8" candidate_uuid: "1f9cb72be98941b5a8e504cbe34132b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f"
I20240501 07:58:59.440433 6576 raft_consensus.cc:3037] T 7d51b1940b594c79a7d13d7026b76da8 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:59.440433 6313 raft_consensus.cc:3037] T 7d51b1940b594c79a7d13d7026b76da8 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:59.441464 6576 raft_consensus.cc:2454] T 7d51b1940b594c79a7d13d7026b76da8 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f9cb72be98941b5a8e504cbe34132b9 in term 1.
I20240501 07:58:59.441475 6313 raft_consensus.cc:2454] T 7d51b1940b594c79a7d13d7026b76da8 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f9cb72be98941b5a8e504cbe34132b9 in term 1.
I20240501 07:58:59.441732 6644 leader_election.cc:302] T 7d51b1940b594c79a7d13d7026b76da8 P 1f9cb72be98941b5a8e504cbe34132b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f9cb72be98941b5a8e504cbe34132b9, 3b164cbbc00f4e8eb3254ddd349a3ff9; no voters:
I20240501 07:58:59.441900 7467 raft_consensus.cc:2781] T 7d51b1940b594c79a7d13d7026b76da8 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:59.441996 7467 raft_consensus.cc:686] T 7d51b1940b594c79a7d13d7026b76da8 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 LEADER]: Becoming Leader. State: Replica: 1f9cb72be98941b5a8e504cbe34132b9, State: Running, Role: LEADER
I20240501 07:58:59.442133 7467 consensus_queue.cc:237] T 7d51b1940b594c79a7d13d7026b76da8 P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } }
I20240501 07:58:59.444108 6167 catalog_manager.cc:5484] T 7d51b1940b594c79a7d13d7026b76da8 P 1f9cb72be98941b5a8e504cbe34132b9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132). New cstate: current_term: 1 leader_uuid: "1f9cb72be98941b5a8e504cbe34132b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } health_report { overall_health: UNKNOWN } } }
I20240501 07:58:59.451148 7306 raft_consensus.cc:482] T 12508ef4d9c942939ce08c14344754b4 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:59.451267 7306 raft_consensus.cc:504] T 12508ef4d9c942939ce08c14344754b4 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } }
I20240501 07:58:59.451664 7306 leader_election.cc:288] T 12508ef4d9c942939ce08c14344754b4 P 8b8b3ad3aa8f4053b2167dd550f2086f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131:34471), 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130:32941)
I20240501 07:58:59.452047 6576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12508ef4d9c942939ce08c14344754b4" candidate_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" is_pre_election: true
I20240501 07:58:59.452111 6446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12508ef4d9c942939ce08c14344754b4" candidate_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8315d22c56f7439a9e1fc82f153f018c" is_pre_election: true
I20240501 07:58:59.452198 6576 raft_consensus.cc:2454] T 12508ef4d9c942939ce08c14344754b4 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b8b3ad3aa8f4053b2167dd550f2086f in term 0.
I20240501 07:58:59.452291 6446 raft_consensus.cc:2454] T 12508ef4d9c942939ce08c14344754b4 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b8b3ad3aa8f4053b2167dd550f2086f in term 0.
I20240501 07:58:59.452572 6251 leader_election.cc:302] T 12508ef4d9c942939ce08c14344754b4 P 8b8b3ad3aa8f4053b2167dd550f2086f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b164cbbc00f4e8eb3254ddd349a3ff9, 8b8b3ad3aa8f4053b2167dd550f2086f; no voters:
I20240501 07:58:59.452772 7306 raft_consensus.cc:2781] T 12508ef4d9c942939ce08c14344754b4 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:59.452832 7306 raft_consensus.cc:482] T 12508ef4d9c942939ce08c14344754b4 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:59.452884 7306 raft_consensus.cc:3037] T 12508ef4d9c942939ce08c14344754b4 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:59.453672 7306 raft_consensus.cc:504] T 12508ef4d9c942939ce08c14344754b4 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } }
I20240501 07:58:59.453976 7306 leader_election.cc:288] T 12508ef4d9c942939ce08c14344754b4 P 8b8b3ad3aa8f4053b2167dd550f2086f [CANDIDATE]: Term 1 election: Requested vote from peers 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131:34471), 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130:32941)
I20240501 07:58:59.454316 6576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12508ef4d9c942939ce08c14344754b4" candidate_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9"
I20240501 07:58:59.454322 6446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12508ef4d9c942939ce08c14344754b4" candidate_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8315d22c56f7439a9e1fc82f153f018c"
I20240501 07:58:59.454424 6576 raft_consensus.cc:3037] T 12508ef4d9c942939ce08c14344754b4 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:59.454442 6446 raft_consensus.cc:3037] T 12508ef4d9c942939ce08c14344754b4 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:59.455133 6576 raft_consensus.cc:2454] T 12508ef4d9c942939ce08c14344754b4 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b8b3ad3aa8f4053b2167dd550f2086f in term 1.
I20240501 07:58:59.455358 6446 raft_consensus.cc:2454] T 12508ef4d9c942939ce08c14344754b4 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b8b3ad3aa8f4053b2167dd550f2086f in term 1.
I20240501 07:58:59.455411 6251 leader_election.cc:302] T 12508ef4d9c942939ce08c14344754b4 P 8b8b3ad3aa8f4053b2167dd550f2086f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b164cbbc00f4e8eb3254ddd349a3ff9, 8b8b3ad3aa8f4053b2167dd550f2086f; no voters:
I20240501 07:58:59.455574 7306 raft_consensus.cc:2781] T 12508ef4d9c942939ce08c14344754b4 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:59.455673 7306 raft_consensus.cc:686] T 12508ef4d9c942939ce08c14344754b4 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 LEADER]: Becoming Leader. State: Replica: 8b8b3ad3aa8f4053b2167dd550f2086f, State: Running, Role: LEADER
I20240501 07:58:59.455821 7306 consensus_queue.cc:237] T 12508ef4d9c942939ce08c14344754b4 P 8b8b3ad3aa8f4053b2167dd550f2086f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } }
I20240501 07:58:59.457651 6167 catalog_manager.cc:5484] T 12508ef4d9c942939ce08c14344754b4 P 8b8b3ad3aa8f4053b2167dd550f2086f reported cstate change: term changed from 0 to 1, leader changed from <none> to 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129). New cstate: current_term: 1 leader_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } health_report { overall_health: UNKNOWN } } }
I20240501 07:58:59.648393 7467 raft_consensus.cc:482] T 0cf117c67b524add96c9e2d887dc9e4d P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:58:59.648548 7467 raft_consensus.cc:504] T 0cf117c67b524add96c9e2d887dc9e4d P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } }
I20240501 07:58:59.649012 7467 leader_election.cc:288] T 0cf117c67b524add96c9e2d887dc9e4d P 1f9cb72be98941b5a8e504cbe34132b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130:32941), 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129:46875)
I20240501 07:58:59.649314 6446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cf117c67b524add96c9e2d887dc9e4d" candidate_uuid: "1f9cb72be98941b5a8e504cbe34132b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8315d22c56f7439a9e1fc82f153f018c" is_pre_election: true
I20240501 07:58:59.649447 6446 raft_consensus.cc:2454] T 0cf117c67b524add96c9e2d887dc9e4d P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f9cb72be98941b5a8e504cbe34132b9 in term 0.
I20240501 07:58:59.649422 6313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cf117c67b524add96c9e2d887dc9e4d" candidate_uuid: "1f9cb72be98941b5a8e504cbe34132b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" is_pre_election: true
I20240501 07:58:59.649597 6313 raft_consensus.cc:2454] T 0cf117c67b524add96c9e2d887dc9e4d P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f9cb72be98941b5a8e504cbe34132b9 in term 0.
I20240501 07:58:59.649703 6643 leader_election.cc:302] T 0cf117c67b524add96c9e2d887dc9e4d P 1f9cb72be98941b5a8e504cbe34132b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f9cb72be98941b5a8e504cbe34132b9, 8315d22c56f7439a9e1fc82f153f018c; no voters:
I20240501 07:58:59.649874 7467 raft_consensus.cc:2781] T 0cf117c67b524add96c9e2d887dc9e4d P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:58:59.649930 7467 raft_consensus.cc:482] T 0cf117c67b524add96c9e2d887dc9e4d P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:58:59.649981 7467 raft_consensus.cc:3037] T 0cf117c67b524add96c9e2d887dc9e4d P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:59.650830 7467 raft_consensus.cc:504] T 0cf117c67b524add96c9e2d887dc9e4d P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } }
I20240501 07:58:59.651103 7467 leader_election.cc:288] T 0cf117c67b524add96c9e2d887dc9e4d P 1f9cb72be98941b5a8e504cbe34132b9 [CANDIDATE]: Term 1 election: Requested vote from peers 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130:32941), 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129:46875)
I20240501 07:58:59.651376 6446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cf117c67b524add96c9e2d887dc9e4d" candidate_uuid: "1f9cb72be98941b5a8e504cbe34132b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8315d22c56f7439a9e1fc82f153f018c"
I20240501 07:58:59.651469 6446 raft_consensus.cc:3037] T 0cf117c67b524add96c9e2d887dc9e4d P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:59.651465 6313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cf117c67b524add96c9e2d887dc9e4d" candidate_uuid: "1f9cb72be98941b5a8e504cbe34132b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f"
I20240501 07:58:59.651614 6313 raft_consensus.cc:3037] T 0cf117c67b524add96c9e2d887dc9e4d P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:58:59.652225 6446 raft_consensus.cc:2454] T 0cf117c67b524add96c9e2d887dc9e4d P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f9cb72be98941b5a8e504cbe34132b9 in term 1.
I20240501 07:58:59.652328 6313 raft_consensus.cc:2454] T 0cf117c67b524add96c9e2d887dc9e4d P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f9cb72be98941b5a8e504cbe34132b9 in term 1.
I20240501 07:58:59.652474 6643 leader_election.cc:302] T 0cf117c67b524add96c9e2d887dc9e4d P 1f9cb72be98941b5a8e504cbe34132b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f9cb72be98941b5a8e504cbe34132b9, 8315d22c56f7439a9e1fc82f153f018c; no voters:
I20240501 07:58:59.652626 7467 raft_consensus.cc:2781] T 0cf117c67b524add96c9e2d887dc9e4d P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:58:59.652702 7467 raft_consensus.cc:686] T 0cf117c67b524add96c9e2d887dc9e4d P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 LEADER]: Becoming Leader. State: Replica: 1f9cb72be98941b5a8e504cbe34132b9, State: Running, Role: LEADER
I20240501 07:58:59.652828 7467 consensus_queue.cc:237] T 0cf117c67b524add96c9e2d887dc9e4d P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } }
I20240501 07:58:59.654645 6167 catalog_manager.cc:5484] T 0cf117c67b524add96c9e2d887dc9e4d P 1f9cb72be98941b5a8e504cbe34132b9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132). New cstate: current_term: 1 leader_uuid: "1f9cb72be98941b5a8e504cbe34132b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } health_report { overall_health: UNKNOWN } } }
W20240501 07:58:59.821054 7916 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:58:59.821314 7916 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:58:59.824628 7916 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_8'
I20240501 07:58:59.849560 7467 consensus_queue.cc:1035] T 7d51b1940b594c79a7d13d7026b76da8 P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:59.853271 7828 consensus_queue.cc:1035] T 7d51b1940b594c79a7d13d7026b76da8 P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:59.920550 6446 raft_consensus.cc:1261] T 0cf117c67b524add96c9e2d887dc9e4d P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Refusing update from remote peer 1f9cb72be98941b5a8e504cbe34132b9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:59.921108 7467 consensus_queue.cc:1035] T 0cf117c67b524add96c9e2d887dc9e4d P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:59.922001 6445 raft_consensus.cc:1261] T 12508ef4d9c942939ce08c14344754b4 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Refusing update from remote peer 8b8b3ad3aa8f4053b2167dd550f2086f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:59.922235 6577 raft_consensus.cc:1261] T 12508ef4d9c942939ce08c14344754b4 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Refusing update from remote peer 8b8b3ad3aa8f4053b2167dd550f2086f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:59.922689 7306 consensus_queue.cc:1035] T 12508ef4d9c942939ce08c14344754b4 P 8b8b3ad3aa8f4053b2167dd550f2086f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:59.922904 7817 consensus_queue.cc:1035] T 12508ef4d9c942939ce08c14344754b4 P 8b8b3ad3aa8f4053b2167dd550f2086f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:58:59.924521 6314 raft_consensus.cc:1261] T 0cf117c67b524add96c9e2d887dc9e4d P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Refusing update from remote peer 1f9cb72be98941b5a8e504cbe34132b9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:58:59.924887 7467 consensus_queue.cc:1035] T 0cf117c67b524add96c9e2d887dc9e4d P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 108.186 ms
time per row: 0.054093 ms
I20240501 07:58:59.983002 6171 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55950:
name: "rebalance_test_table_9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 07:58:59.988476 6424 tablet_service.cc:1467] Processing CreateTablet for tablet 72feac2a993141168ac9399d9361ef17 (DEFAULT_TABLE table=rebalance_test_table_9 [id=c4f46595186f4427aa00d6752aa5dc9a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:59.988811 6424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72feac2a993141168ac9399d9361ef17. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:59.988740 6688 tablet_service.cc:1467] Processing CreateTablet for tablet 72feac2a993141168ac9399d9361ef17 (DEFAULT_TABLE table=rebalance_test_table_9 [id=c4f46595186f4427aa00d6752aa5dc9a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:59.989077 6688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72feac2a993141168ac9399d9361ef17. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:59.989615 6556 tablet_service.cc:1467] Processing CreateTablet for tablet 72feac2a993141168ac9399d9361ef17 (DEFAULT_TABLE table=rebalance_test_table_9 [id=c4f46595186f4427aa00d6752aa5dc9a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:59.989689 6557 tablet_service.cc:1467] Processing CreateTablet for tablet a756205204d146ce82a2799e30582f57 (DEFAULT_TABLE table=rebalance_test_table_9 [id=c4f46595186f4427aa00d6752aa5dc9a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:59.989869 6556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72feac2a993141168ac9399d9361ef17. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:59.990540 6426 tablet_service.cc:1467] Processing CreateTablet for tablet a756205204d146ce82a2799e30582f57 (DEFAULT_TABLE table=rebalance_test_table_9 [id=c4f46595186f4427aa00d6752aa5dc9a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:59.990756 6426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a756205204d146ce82a2799e30582f57. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:59.991431 6555 tablet_service.cc:1467] Processing CreateTablet for tablet 6dc929e2a6ae45bd8b1f297583636bb6 (DEFAULT_TABLE table=rebalance_test_table_9 [id=c4f46595186f4427aa00d6752aa5dc9a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:59.991685 6555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6dc929e2a6ae45bd8b1f297583636bb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:59.992172 7979 tablet_bootstrap.cc:492] T 72feac2a993141168ac9399d9361ef17 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Bootstrap starting.
I20240501 07:58:59.992327 7977 tablet_bootstrap.cc:492] T 72feac2a993141168ac9399d9361ef17 P 8315d22c56f7439a9e1fc82f153f018c: Bootstrap starting.
I20240501 07:58:59.992780 6425 tablet_service.cc:1467] Processing CreateTablet for tablet 6dc929e2a6ae45bd8b1f297583636bb6 (DEFAULT_TABLE table=rebalance_test_table_9 [id=c4f46595186f4427aa00d6752aa5dc9a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:59.993033 6425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6dc929e2a6ae45bd8b1f297583636bb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:59.993281 7979 tablet_bootstrap.cc:654] T 72feac2a993141168ac9399d9361ef17 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:59.994288 7977 tablet_bootstrap.cc:654] T 72feac2a993141168ac9399d9361ef17 P 8315d22c56f7439a9e1fc82f153f018c: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:59.994868 6557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a756205204d146ce82a2799e30582f57. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:59.994892 7978 tablet_bootstrap.cc:492] T 72feac2a993141168ac9399d9361ef17 P 1f9cb72be98941b5a8e504cbe34132b9: Bootstrap starting.
I20240501 07:58:59.995257 7977 tablet_bootstrap.cc:492] T 72feac2a993141168ac9399d9361ef17 P 8315d22c56f7439a9e1fc82f153f018c: No bootstrap required, opened a new log
I20240501 07:58:59.995335 7977 ts_tablet_manager.cc:1367] T 72feac2a993141168ac9399d9361ef17 P 8315d22c56f7439a9e1fc82f153f018c: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240501 07:58:59.995704 7977 raft_consensus.cc:348] T 72feac2a993141168ac9399d9361ef17 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } }
I20240501 07:58:59.995793 7977 raft_consensus.cc:374] T 72feac2a993141168ac9399d9361ef17 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:58:59.995827 7977 raft_consensus.cc:729] T 72feac2a993141168ac9399d9361ef17 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8315d22c56f7439a9e1fc82f153f018c, State: Initialized, Role: FOLLOWER
I20240501 07:58:59.995955 7977 consensus_queue.cc:260] T 72feac2a993141168ac9399d9361ef17 P 8315d22c56f7439a9e1fc82f153f018c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } }
I20240501 07:58:59.996201 7977 ts_tablet_manager.cc:1398] T 72feac2a993141168ac9399d9361ef17 P 8315d22c56f7439a9e1fc82f153f018c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:58:59.997026 7977 tablet_bootstrap.cc:492] T a756205204d146ce82a2799e30582f57 P 8315d22c56f7439a9e1fc82f153f018c: Bootstrap starting.
I20240501 07:58:59.997226 7978 tablet_bootstrap.cc:654] T 72feac2a993141168ac9399d9361ef17 P 1f9cb72be98941b5a8e504cbe34132b9: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:59.997224 6688 tablet_service.cc:1467] Processing CreateTablet for tablet a756205204d146ce82a2799e30582f57 (DEFAULT_TABLE table=rebalance_test_table_9 [id=c4f46595186f4427aa00d6752aa5dc9a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:58:59.997498 6688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a756205204d146ce82a2799e30582f57. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:58:59.998871 7977 tablet_bootstrap.cc:654] T a756205204d146ce82a2799e30582f57 P 8315d22c56f7439a9e1fc82f153f018c: Neither blocks nor log segments found. Creating new log.
I20240501 07:58:59.999985 7977 tablet_bootstrap.cc:492] T a756205204d146ce82a2799e30582f57 P 8315d22c56f7439a9e1fc82f153f018c: No bootstrap required, opened a new log
I20240501 07:59:00.000073 7977 ts_tablet_manager.cc:1367] T a756205204d146ce82a2799e30582f57 P 8315d22c56f7439a9e1fc82f153f018c: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 07:59:00.000259 6294 tablet_service.cc:1467] Processing CreateTablet for tablet 6dc929e2a6ae45bd8b1f297583636bb6 (DEFAULT_TABLE table=rebalance_test_table_9 [id=c4f46595186f4427aa00d6752aa5dc9a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:00.000535 6294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6dc929e2a6ae45bd8b1f297583636bb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:00.000586 7977 raft_consensus.cc:348] T a756205204d146ce82a2799e30582f57 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } }
I20240501 07:59:00.000666 7977 raft_consensus.cc:374] T a756205204d146ce82a2799e30582f57 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:00.000687 7977 raft_consensus.cc:729] T a756205204d146ce82a2799e30582f57 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8315d22c56f7439a9e1fc82f153f018c, State: Initialized, Role: FOLLOWER
I20240501 07:59:00.000692 7979 tablet_bootstrap.cc:492] T 72feac2a993141168ac9399d9361ef17 P 3b164cbbc00f4e8eb3254ddd349a3ff9: No bootstrap required, opened a new log
I20240501 07:59:00.000771 7979 ts_tablet_manager.cc:1367] T 72feac2a993141168ac9399d9361ef17 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.002s
I20240501 07:59:00.000797 7977 consensus_queue.cc:260] T a756205204d146ce82a2799e30582f57 P 8315d22c56f7439a9e1fc82f153f018c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } }
I20240501 07:59:00.001114 7977 ts_tablet_manager.cc:1398] T a756205204d146ce82a2799e30582f57 P 8315d22c56f7439a9e1fc82f153f018c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:00.001242 7977 tablet_bootstrap.cc:492] T 6dc929e2a6ae45bd8b1f297583636bb6 P 8315d22c56f7439a9e1fc82f153f018c: Bootstrap starting.
I20240501 07:59:00.001276 7979 raft_consensus.cc:348] T 72feac2a993141168ac9399d9361ef17 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } }
I20240501 07:59:00.001371 7979 raft_consensus.cc:374] T 72feac2a993141168ac9399d9361ef17 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:00.001372 7978 tablet_bootstrap.cc:492] T 72feac2a993141168ac9399d9361ef17 P 1f9cb72be98941b5a8e504cbe34132b9: No bootstrap required, opened a new log
I20240501 07:59:00.001405 7979 raft_consensus.cc:729] T 72feac2a993141168ac9399d9361ef17 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b164cbbc00f4e8eb3254ddd349a3ff9, State: Initialized, Role: FOLLOWER
I20240501 07:59:00.001469 7978 ts_tablet_manager.cc:1367] T 72feac2a993141168ac9399d9361ef17 P 1f9cb72be98941b5a8e504cbe34132b9: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240501 07:59:00.001529 7979 consensus_queue.cc:260] T 72feac2a993141168ac9399d9361ef17 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } }
I20240501 07:59:00.001821 7979 ts_tablet_manager.cc:1398] T 72feac2a993141168ac9399d9361ef17 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:59:00.001982 7977 tablet_bootstrap.cc:654] T 6dc929e2a6ae45bd8b1f297583636bb6 P 8315d22c56f7439a9e1fc82f153f018c: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:00.001993 7979 tablet_bootstrap.cc:492] T 6dc929e2a6ae45bd8b1f297583636bb6 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Bootstrap starting.
I20240501 07:59:00.002099 7978 raft_consensus.cc:348] T 72feac2a993141168ac9399d9361ef17 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } }
I20240501 07:59:00.002192 7978 raft_consensus.cc:374] T 72feac2a993141168ac9399d9361ef17 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:00.002223 7978 raft_consensus.cc:729] T 72feac2a993141168ac9399d9361ef17 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f9cb72be98941b5a8e504cbe34132b9, State: Initialized, Role: FOLLOWER
I20240501 07:59:00.002336 7978 consensus_queue.cc:260] T 72feac2a993141168ac9399d9361ef17 P 1f9cb72be98941b5a8e504cbe34132b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } }
I20240501 07:59:00.002621 7978 ts_tablet_manager.cc:1398] T 72feac2a993141168ac9399d9361ef17 P 1f9cb72be98941b5a8e504cbe34132b9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:00.002781 7978 tablet_bootstrap.cc:492] T a756205204d146ce82a2799e30582f57 P 1f9cb72be98941b5a8e504cbe34132b9: Bootstrap starting.
I20240501 07:59:00.002880 7979 tablet_bootstrap.cc:654] T 6dc929e2a6ae45bd8b1f297583636bb6 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:00.002990 7977 tablet_bootstrap.cc:492] T 6dc929e2a6ae45bd8b1f297583636bb6 P 8315d22c56f7439a9e1fc82f153f018c: No bootstrap required, opened a new log
I20240501 07:59:00.003046 7977 ts_tablet_manager.cc:1367] T 6dc929e2a6ae45bd8b1f297583636bb6 P 8315d22c56f7439a9e1fc82f153f018c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:59:00.003330 7977 raft_consensus.cc:348] T 6dc929e2a6ae45bd8b1f297583636bb6 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } }
I20240501 07:59:00.003435 7977 raft_consensus.cc:374] T 6dc929e2a6ae45bd8b1f297583636bb6 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:00.003466 7977 raft_consensus.cc:729] T 6dc929e2a6ae45bd8b1f297583636bb6 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8315d22c56f7439a9e1fc82f153f018c, State: Initialized, Role: FOLLOWER
I20240501 07:59:00.003585 7977 consensus_queue.cc:260] T 6dc929e2a6ae45bd8b1f297583636bb6 P 8315d22c56f7439a9e1fc82f153f018c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } }
I20240501 07:59:00.003688 7978 tablet_bootstrap.cc:654] T a756205204d146ce82a2799e30582f57 P 1f9cb72be98941b5a8e504cbe34132b9: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:00.003856 7977 ts_tablet_manager.cc:1398] T 6dc929e2a6ae45bd8b1f297583636bb6 P 8315d22c56f7439a9e1fc82f153f018c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:59:00.004741 7978 tablet_bootstrap.cc:492] T a756205204d146ce82a2799e30582f57 P 1f9cb72be98941b5a8e504cbe34132b9: No bootstrap required, opened a new log
I20240501 07:59:00.004825 7978 ts_tablet_manager.cc:1367] T a756205204d146ce82a2799e30582f57 P 1f9cb72be98941b5a8e504cbe34132b9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:59:00.005149 7979 tablet_bootstrap.cc:492] T 6dc929e2a6ae45bd8b1f297583636bb6 P 3b164cbbc00f4e8eb3254ddd349a3ff9: No bootstrap required, opened a new log
I20240501 07:59:00.005223 7979 ts_tablet_manager.cc:1367] T 6dc929e2a6ae45bd8b1f297583636bb6 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240501 07:59:00.005285 7978 raft_consensus.cc:348] T a756205204d146ce82a2799e30582f57 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } }
I20240501 07:59:00.005378 7978 raft_consensus.cc:374] T a756205204d146ce82a2799e30582f57 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:00.005406 7978 raft_consensus.cc:729] T a756205204d146ce82a2799e30582f57 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f9cb72be98941b5a8e504cbe34132b9, State: Initialized, Role: FOLLOWER
I20240501 07:59:00.005580 7979 raft_consensus.cc:348] T 6dc929e2a6ae45bd8b1f297583636bb6 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } }
I20240501 07:59:00.005664 7979 raft_consensus.cc:374] T 6dc929e2a6ae45bd8b1f297583636bb6 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:00.005692 7979 raft_consensus.cc:729] T 6dc929e2a6ae45bd8b1f297583636bb6 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b164cbbc00f4e8eb3254ddd349a3ff9, State: Initialized, Role: FOLLOWER
I20240501 07:59:00.005795 7979 consensus_queue.cc:260] T 6dc929e2a6ae45bd8b1f297583636bb6 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } }
I20240501 07:59:00.006729 7979 ts_tablet_manager.cc:1398] T 6dc929e2a6ae45bd8b1f297583636bb6 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:00.006889 7979 tablet_bootstrap.cc:492] T a756205204d146ce82a2799e30582f57 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Bootstrap starting.
I20240501 07:59:00.007690 7979 tablet_bootstrap.cc:654] T a756205204d146ce82a2799e30582f57 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:00.008013 7978 consensus_queue.cc:260] T a756205204d146ce82a2799e30582f57 P 1f9cb72be98941b5a8e504cbe34132b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } }
I20240501 07:59:00.008383 7978 ts_tablet_manager.cc:1398] T a756205204d146ce82a2799e30582f57 P 1f9cb72be98941b5a8e504cbe34132b9: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20240501 07:59:00.009073 7988 tablet_bootstrap.cc:492] T 6dc929e2a6ae45bd8b1f297583636bb6 P 8b8b3ad3aa8f4053b2167dd550f2086f: Bootstrap starting.
I20240501 07:59:00.009977 7988 tablet_bootstrap.cc:654] T 6dc929e2a6ae45bd8b1f297583636bb6 P 8b8b3ad3aa8f4053b2167dd550f2086f: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:00.010941 7988 tablet_bootstrap.cc:492] T 6dc929e2a6ae45bd8b1f297583636bb6 P 8b8b3ad3aa8f4053b2167dd550f2086f: No bootstrap required, opened a new log
I20240501 07:59:00.011003 7988 ts_tablet_manager.cc:1367] T 6dc929e2a6ae45bd8b1f297583636bb6 P 8b8b3ad3aa8f4053b2167dd550f2086f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:59:00.011389 7988 raft_consensus.cc:348] T 6dc929e2a6ae45bd8b1f297583636bb6 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } }
I20240501 07:59:00.011474 7988 raft_consensus.cc:374] T 6dc929e2a6ae45bd8b1f297583636bb6 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:00.011502 7988 raft_consensus.cc:729] T 6dc929e2a6ae45bd8b1f297583636bb6 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b8b3ad3aa8f4053b2167dd550f2086f, State: Initialized, Role: FOLLOWER
I20240501 07:59:00.011602 7988 consensus_queue.cc:260] T 6dc929e2a6ae45bd8b1f297583636bb6 P 8b8b3ad3aa8f4053b2167dd550f2086f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } }
I20240501 07:59:00.011821 7988 ts_tablet_manager.cc:1398] T 6dc929e2a6ae45bd8b1f297583636bb6 P 8b8b3ad3aa8f4053b2167dd550f2086f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:00.024765 7979 tablet_bootstrap.cc:492] T a756205204d146ce82a2799e30582f57 P 3b164cbbc00f4e8eb3254ddd349a3ff9: No bootstrap required, opened a new log
I20240501 07:59:00.024888 7979 ts_tablet_manager.cc:1367] T a756205204d146ce82a2799e30582f57 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Time spent bootstrapping tablet: real 0.018s user 0.001s sys 0.000s
I20240501 07:59:00.025422 7979 raft_consensus.cc:348] T a756205204d146ce82a2799e30582f57 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } }
I20240501 07:59:00.025544 7979 raft_consensus.cc:374] T a756205204d146ce82a2799e30582f57 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:00.025573 7979 raft_consensus.cc:729] T a756205204d146ce82a2799e30582f57 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b164cbbc00f4e8eb3254ddd349a3ff9, State: Initialized, Role: FOLLOWER
I20240501 07:59:00.025692 7979 consensus_queue.cc:260] T a756205204d146ce82a2799e30582f57 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } }
I20240501 07:59:00.025959 7979 ts_tablet_manager.cc:1398] T a756205204d146ce82a2799e30582f57 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:00.049605 7959 raft_consensus.cc:482] T 72feac2a993141168ac9399d9361ef17 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:00.049744 7959 raft_consensus.cc:504] T 72feac2a993141168ac9399d9361ef17 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } }
I20240501 07:59:00.050164 7959 leader_election.cc:288] T 72feac2a993141168ac9399d9361ef17 P 1f9cb72be98941b5a8e504cbe34132b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130:32941), 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131:34471)
I20240501 07:59:00.050592 6577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72feac2a993141168ac9399d9361ef17" candidate_uuid: "1f9cb72be98941b5a8e504cbe34132b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" is_pre_election: true
I20240501 07:59:00.050591 6445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72feac2a993141168ac9399d9361ef17" candidate_uuid: "1f9cb72be98941b5a8e504cbe34132b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8315d22c56f7439a9e1fc82f153f018c" is_pre_election: true
I20240501 07:59:00.050740 6445 raft_consensus.cc:2454] T 72feac2a993141168ac9399d9361ef17 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f9cb72be98941b5a8e504cbe34132b9 in term 0.
I20240501 07:59:00.050740 6577 raft_consensus.cc:2454] T 72feac2a993141168ac9399d9361ef17 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f9cb72be98941b5a8e504cbe34132b9 in term 0.
I20240501 07:59:00.051035 6644 leader_election.cc:302] T 72feac2a993141168ac9399d9361ef17 P 1f9cb72be98941b5a8e504cbe34132b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f9cb72be98941b5a8e504cbe34132b9, 3b164cbbc00f4e8eb3254ddd349a3ff9; no voters:
I20240501 07:59:00.051214 7959 raft_consensus.cc:2781] T 72feac2a993141168ac9399d9361ef17 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:00.051290 7959 raft_consensus.cc:482] T 72feac2a993141168ac9399d9361ef17 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:00.051344 7959 raft_consensus.cc:3037] T 72feac2a993141168ac9399d9361ef17 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:00.052258 7959 raft_consensus.cc:504] T 72feac2a993141168ac9399d9361ef17 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } }
I20240501 07:59:00.052575 7959 leader_election.cc:288] T 72feac2a993141168ac9399d9361ef17 P 1f9cb72be98941b5a8e504cbe34132b9 [CANDIDATE]: Term 1 election: Requested vote from peers 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130:32941), 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131:34471)
I20240501 07:59:00.052894 6445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72feac2a993141168ac9399d9361ef17" candidate_uuid: "1f9cb72be98941b5a8e504cbe34132b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8315d22c56f7439a9e1fc82f153f018c"
I20240501 07:59:00.052932 6577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72feac2a993141168ac9399d9361ef17" candidate_uuid: "1f9cb72be98941b5a8e504cbe34132b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9"
I20240501 07:59:00.053061 6445 raft_consensus.cc:3037] T 72feac2a993141168ac9399d9361ef17 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:00.053092 6577 raft_consensus.cc:3037] T 72feac2a993141168ac9399d9361ef17 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:00.053838 6445 raft_consensus.cc:2454] T 72feac2a993141168ac9399d9361ef17 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f9cb72be98941b5a8e504cbe34132b9 in term 1.
I20240501 07:59:00.053880 6577 raft_consensus.cc:2454] T 72feac2a993141168ac9399d9361ef17 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f9cb72be98941b5a8e504cbe34132b9 in term 1.
I20240501 07:59:00.054116 6643 leader_election.cc:302] T 72feac2a993141168ac9399d9361ef17 P 1f9cb72be98941b5a8e504cbe34132b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f9cb72be98941b5a8e504cbe34132b9, 8315d22c56f7439a9e1fc82f153f018c; no voters:
I20240501 07:59:00.054301 7959 raft_consensus.cc:2781] T 72feac2a993141168ac9399d9361ef17 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:00.054378 7959 raft_consensus.cc:686] T 72feac2a993141168ac9399d9361ef17 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 LEADER]: Becoming Leader. State: Replica: 1f9cb72be98941b5a8e504cbe34132b9, State: Running, Role: LEADER
I20240501 07:59:00.054531 7959 consensus_queue.cc:237] T 72feac2a993141168ac9399d9361ef17 P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } }
I20240501 07:59:00.056180 6165 catalog_manager.cc:5484] T 72feac2a993141168ac9399d9361ef17 P 1f9cb72be98941b5a8e504cbe34132b9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132). New cstate: current_term: 1 leader_uuid: "1f9cb72be98941b5a8e504cbe34132b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } health_report { overall_health: UNKNOWN } } }
I20240501 07:59:00.164134 7758 raft_consensus.cc:482] T a756205204d146ce82a2799e30582f57 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:00.164263 7758 raft_consensus.cc:504] T a756205204d146ce82a2799e30582f57 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } }
I20240501 07:59:00.164659 7758 leader_election.cc:288] T a756205204d146ce82a2799e30582f57 P 8315d22c56f7439a9e1fc82f153f018c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132:36773), 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131:34471)
I20240501 07:59:00.165040 6577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a756205204d146ce82a2799e30582f57" candidate_uuid: "8315d22c56f7439a9e1fc82f153f018c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" is_pre_election: true
I20240501 07:59:00.165032 6708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a756205204d146ce82a2799e30582f57" candidate_uuid: "8315d22c56f7439a9e1fc82f153f018c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f9cb72be98941b5a8e504cbe34132b9" is_pre_election: true
I20240501 07:59:00.165187 6708 raft_consensus.cc:2454] T a756205204d146ce82a2799e30582f57 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8315d22c56f7439a9e1fc82f153f018c in term 0.
I20240501 07:59:00.165186 6577 raft_consensus.cc:2454] T a756205204d146ce82a2799e30582f57 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8315d22c56f7439a9e1fc82f153f018c in term 0.
I20240501 07:59:00.165485 6382 leader_election.cc:302] T a756205204d146ce82a2799e30582f57 P 8315d22c56f7439a9e1fc82f153f018c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b164cbbc00f4e8eb3254ddd349a3ff9, 8315d22c56f7439a9e1fc82f153f018c; no voters:
I20240501 07:59:00.165663 7758 raft_consensus.cc:2781] T a756205204d146ce82a2799e30582f57 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:00.165731 7758 raft_consensus.cc:482] T a756205204d146ce82a2799e30582f57 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:00.165762 7758 raft_consensus.cc:3037] T a756205204d146ce82a2799e30582f57 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:00.166623 7758 raft_consensus.cc:504] T a756205204d146ce82a2799e30582f57 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } }
I20240501 07:59:00.166929 7758 leader_election.cc:288] T a756205204d146ce82a2799e30582f57 P 8315d22c56f7439a9e1fc82f153f018c [CANDIDATE]: Term 1 election: Requested vote from peers 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132:36773), 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131:34471)
I20240501 07:59:00.167246 6708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a756205204d146ce82a2799e30582f57" candidate_uuid: "8315d22c56f7439a9e1fc82f153f018c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f9cb72be98941b5a8e504cbe34132b9"
I20240501 07:59:00.167254 6577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a756205204d146ce82a2799e30582f57" candidate_uuid: "8315d22c56f7439a9e1fc82f153f018c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9"
I20240501 07:59:00.167384 6708 raft_consensus.cc:3037] T a756205204d146ce82a2799e30582f57 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:00.167403 6577 raft_consensus.cc:3037] T a756205204d146ce82a2799e30582f57 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:00.168318 6577 raft_consensus.cc:2454] T a756205204d146ce82a2799e30582f57 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8315d22c56f7439a9e1fc82f153f018c in term 1.
I20240501 07:59:00.168319 6708 raft_consensus.cc:2454] T a756205204d146ce82a2799e30582f57 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8315d22c56f7439a9e1fc82f153f018c in term 1.
I20240501 07:59:00.168606 6381 leader_election.cc:302] T a756205204d146ce82a2799e30582f57 P 8315d22c56f7439a9e1fc82f153f018c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f9cb72be98941b5a8e504cbe34132b9, 8315d22c56f7439a9e1fc82f153f018c; no voters:
I20240501 07:59:00.168802 7758 raft_consensus.cc:2781] T a756205204d146ce82a2799e30582f57 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:00.168871 7758 raft_consensus.cc:686] T a756205204d146ce82a2799e30582f57 P 8315d22c56f7439a9e1fc82f153f018c [term 1 LEADER]: Becoming Leader. State: Replica: 8315d22c56f7439a9e1fc82f153f018c, State: Running, Role: LEADER
I20240501 07:59:00.169052 7758 consensus_queue.cc:237] T a756205204d146ce82a2799e30582f57 P 8315d22c56f7439a9e1fc82f153f018c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } }
I20240501 07:59:00.170900 6165 catalog_manager.cc:5484] T a756205204d146ce82a2799e30582f57 P 8315d22c56f7439a9e1fc82f153f018c reported cstate change: term changed from 0 to 1, leader changed from <none> to 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130). New cstate: current_term: 1 leader_uuid: "8315d22c56f7439a9e1fc82f153f018c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } health_report { overall_health: UNKNOWN } } }
I20240501 07:59:00.348358 7758 raft_consensus.cc:482] T 6dc929e2a6ae45bd8b1f297583636bb6 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:00.348487 7758 raft_consensus.cc:504] T 6dc929e2a6ae45bd8b1f297583636bb6 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } }
I20240501 07:59:00.348887 7758 leader_election.cc:288] T 6dc929e2a6ae45bd8b1f297583636bb6 P 8315d22c56f7439a9e1fc82f153f018c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131:34471), 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129:46875)
I20240501 07:59:00.349282 6313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6dc929e2a6ae45bd8b1f297583636bb6" candidate_uuid: "8315d22c56f7439a9e1fc82f153f018c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" is_pre_election: true
I20240501 07:59:00.349285 6576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6dc929e2a6ae45bd8b1f297583636bb6" candidate_uuid: "8315d22c56f7439a9e1fc82f153f018c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" is_pre_election: true
I20240501 07:59:00.349440 6313 raft_consensus.cc:2454] T 6dc929e2a6ae45bd8b1f297583636bb6 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8315d22c56f7439a9e1fc82f153f018c in term 0.
I20240501 07:59:00.349442 6576 raft_consensus.cc:2454] T 6dc929e2a6ae45bd8b1f297583636bb6 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8315d22c56f7439a9e1fc82f153f018c in term 0.
I20240501 07:59:00.349735 6382 leader_election.cc:302] T 6dc929e2a6ae45bd8b1f297583636bb6 P 8315d22c56f7439a9e1fc82f153f018c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b164cbbc00f4e8eb3254ddd349a3ff9, 8315d22c56f7439a9e1fc82f153f018c; no voters:
I20240501 07:59:00.349972 7758 raft_consensus.cc:2781] T 6dc929e2a6ae45bd8b1f297583636bb6 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:00.350033 7758 raft_consensus.cc:482] T 6dc929e2a6ae45bd8b1f297583636bb6 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:00.350077 7758 raft_consensus.cc:3037] T 6dc929e2a6ae45bd8b1f297583636bb6 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:00.350905 7758 raft_consensus.cc:504] T 6dc929e2a6ae45bd8b1f297583636bb6 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } }
I20240501 07:59:00.351234 7758 leader_election.cc:288] T 6dc929e2a6ae45bd8b1f297583636bb6 P 8315d22c56f7439a9e1fc82f153f018c [CANDIDATE]: Term 1 election: Requested vote from peers 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131:34471), 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129:46875)
I20240501 07:59:00.351595 6576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6dc929e2a6ae45bd8b1f297583636bb6" candidate_uuid: "8315d22c56f7439a9e1fc82f153f018c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9"
I20240501 07:59:00.351711 6576 raft_consensus.cc:3037] T 6dc929e2a6ae45bd8b1f297583636bb6 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:00.351831 6314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6dc929e2a6ae45bd8b1f297583636bb6" candidate_uuid: "8315d22c56f7439a9e1fc82f153f018c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f"
I20240501 07:59:00.351949 6314 raft_consensus.cc:3037] T 6dc929e2a6ae45bd8b1f297583636bb6 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:00.352547 6576 raft_consensus.cc:2454] T 6dc929e2a6ae45bd8b1f297583636bb6 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8315d22c56f7439a9e1fc82f153f018c in term 1.
I20240501 07:59:00.352682 6314 raft_consensus.cc:2454] T 6dc929e2a6ae45bd8b1f297583636bb6 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8315d22c56f7439a9e1fc82f153f018c in term 1.
I20240501 07:59:00.352840 6382 leader_election.cc:302] T 6dc929e2a6ae45bd8b1f297583636bb6 P 8315d22c56f7439a9e1fc82f153f018c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b164cbbc00f4e8eb3254ddd349a3ff9, 8315d22c56f7439a9e1fc82f153f018c; no voters:
I20240501 07:59:00.353080 7758 raft_consensus.cc:2781] T 6dc929e2a6ae45bd8b1f297583636bb6 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:00.353161 7758 raft_consensus.cc:686] T 6dc929e2a6ae45bd8b1f297583636bb6 P 8315d22c56f7439a9e1fc82f153f018c [term 1 LEADER]: Becoming Leader. State: Replica: 8315d22c56f7439a9e1fc82f153f018c, State: Running, Role: LEADER
I20240501 07:59:00.353302 7758 consensus_queue.cc:237] T 6dc929e2a6ae45bd8b1f297583636bb6 P 8315d22c56f7439a9e1fc82f153f018c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } }
I20240501 07:59:00.354977 6165 catalog_manager.cc:5484] T 6dc929e2a6ae45bd8b1f297583636bb6 P 8315d22c56f7439a9e1fc82f153f018c reported cstate change: term changed from 0 to 1, leader changed from <none> to 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130). New cstate: current_term: 1 leader_uuid: "8315d22c56f7439a9e1fc82f153f018c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } health_report { overall_health: HEALTHY } } }
I20240501 07:59:00.462149 7959 consensus_queue.cc:1035] T 72feac2a993141168ac9399d9361ef17 P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:00.466135 7963 consensus_queue.cc:1035] T 72feac2a993141168ac9399d9361ef17 P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 07:59:00.491322 7993 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:59:00.491638 7993 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:59:00.495030 7993 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_9'
I20240501 07:59:00.545993 7954 consensus_queue.cc:1035] T a756205204d146ce82a2799e30582f57 P 8315d22c56f7439a9e1fc82f153f018c [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:00.553576 7758 consensus_queue.cc:1035] T a756205204d146ce82a2799e30582f57 P 8315d22c56f7439a9e1fc82f153f018c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:00.587337 6577 raft_consensus.cc:1261] T 6dc929e2a6ae45bd8b1f297583636bb6 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Refusing update from remote peer 8315d22c56f7439a9e1fc82f153f018c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:00.587774 6314 raft_consensus.cc:1261] T 6dc929e2a6ae45bd8b1f297583636bb6 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Refusing update from remote peer 8315d22c56f7439a9e1fc82f153f018c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:00.590528 7758 consensus_queue.cc:1035] T 6dc929e2a6ae45bd8b1f297583636bb6 P 8315d22c56f7439a9e1fc82f153f018c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:00.591152 7758 consensus_queue.cc:1035] T 6dc929e2a6ae45bd8b1f297583636bb6 P 8315d22c56f7439a9e1fc82f153f018c [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 119.126 ms
time per row: 0.0595629 ms
I20240501 07:59:00.665446 6168 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55950:
name: "rebalance_test_table_10"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 07:59:00.670835 6294 tablet_service.cc:1467] Processing CreateTablet for tablet f9df6f3df2b94e5eb97dbff04dbe2ae1 (DEFAULT_TABLE table=rebalance_test_table_10 [id=7b24d2d13db34edb849642d7d3eef300]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:00.670840 6425 tablet_service.cc:1467] Processing CreateTablet for tablet f9df6f3df2b94e5eb97dbff04dbe2ae1 (DEFAULT_TABLE table=rebalance_test_table_10 [id=7b24d2d13db34edb849642d7d3eef300]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:00.671164 6294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9df6f3df2b94e5eb97dbff04dbe2ae1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:00.671161 6425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9df6f3df2b94e5eb97dbff04dbe2ae1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:00.671345 6687 tablet_service.cc:1467] Processing CreateTablet for tablet 187940ba79f546b1ad79e7ec1de76442 (DEFAULT_TABLE table=rebalance_test_table_10 [id=7b24d2d13db34edb849642d7d3eef300]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:00.671413 6688 tablet_service.cc:1467] Processing CreateTablet for tablet f9df6f3df2b94e5eb97dbff04dbe2ae1 (DEFAULT_TABLE table=rebalance_test_table_10 [id=7b24d2d13db34edb849642d7d3eef300]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:00.672134 6686 tablet_service.cc:1467] Processing CreateTablet for tablet 5122bc9a3cfb4b04aafc739256914bc2 (DEFAULT_TABLE table=rebalance_test_table_10 [id=7b24d2d13db34edb849642d7d3eef300]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:00.672176 6426 tablet_service.cc:1467] Processing CreateTablet for tablet 5122bc9a3cfb4b04aafc739256914bc2 (DEFAULT_TABLE table=rebalance_test_table_10 [id=7b24d2d13db34edb849642d7d3eef300]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:00.672408 6426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5122bc9a3cfb4b04aafc739256914bc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:00.672454 6687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 187940ba79f546b1ad79e7ec1de76442. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:00.673146 6688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9df6f3df2b94e5eb97dbff04dbe2ae1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:00.673966 6294 tablet_service.cc:1467] Processing CreateTablet for tablet 187940ba79f546b1ad79e7ec1de76442 (DEFAULT_TABLE table=rebalance_test_table_10 [id=7b24d2d13db34edb849642d7d3eef300]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:00.673950 6557 tablet_service.cc:1467] Processing CreateTablet for tablet 187940ba79f546b1ad79e7ec1de76442 (DEFAULT_TABLE table=rebalance_test_table_10 [id=7b24d2d13db34edb849642d7d3eef300]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:00.674188 6557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 187940ba79f546b1ad79e7ec1de76442. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:00.674188 6294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 187940ba79f546b1ad79e7ec1de76442. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:00.675254 8052 tablet_bootstrap.cc:492] T f9df6f3df2b94e5eb97dbff04dbe2ae1 P 1f9cb72be98941b5a8e504cbe34132b9: Bootstrap starting.
I20240501 07:59:00.675555 6686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5122bc9a3cfb4b04aafc739256914bc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:00.676194 8051 tablet_bootstrap.cc:492] T f9df6f3df2b94e5eb97dbff04dbe2ae1 P 8b8b3ad3aa8f4053b2167dd550f2086f: Bootstrap starting.
I20240501 07:59:00.676478 8052 tablet_bootstrap.cc:654] T f9df6f3df2b94e5eb97dbff04dbe2ae1 P 1f9cb72be98941b5a8e504cbe34132b9: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:00.676508 6293 tablet_service.cc:1467] Processing CreateTablet for tablet 5122bc9a3cfb4b04aafc739256914bc2 (DEFAULT_TABLE table=rebalance_test_table_10 [id=7b24d2d13db34edb849642d7d3eef300]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:00.676743 6293 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5122bc9a3cfb4b04aafc739256914bc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:00.677080 8051 tablet_bootstrap.cc:654] T f9df6f3df2b94e5eb97dbff04dbe2ae1 P 8b8b3ad3aa8f4053b2167dd550f2086f: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:00.677752 8050 tablet_bootstrap.cc:492] T f9df6f3df2b94e5eb97dbff04dbe2ae1 P 8315d22c56f7439a9e1fc82f153f018c: Bootstrap starting.
I20240501 07:59:00.678198 8051 tablet_bootstrap.cc:492] T f9df6f3df2b94e5eb97dbff04dbe2ae1 P 8b8b3ad3aa8f4053b2167dd550f2086f: No bootstrap required, opened a new log
I20240501 07:59:00.678292 8051 ts_tablet_manager.cc:1367] T f9df6f3df2b94e5eb97dbff04dbe2ae1 P 8b8b3ad3aa8f4053b2167dd550f2086f: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240501 07:59:00.678697 8051 raft_consensus.cc:348] T f9df6f3df2b94e5eb97dbff04dbe2ae1 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:59:00.678822 8051 raft_consensus.cc:374] T f9df6f3df2b94e5eb97dbff04dbe2ae1 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:00.679272 8053 tablet_bootstrap.cc:492] T 187940ba79f546b1ad79e7ec1de76442 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Bootstrap starting.
I20240501 07:59:00.679698 8050 tablet_bootstrap.cc:654] T f9df6f3df2b94e5eb97dbff04dbe2ae1 P 8315d22c56f7439a9e1fc82f153f018c: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:00.679828 8052 tablet_bootstrap.cc:492] T f9df6f3df2b94e5eb97dbff04dbe2ae1 P 1f9cb72be98941b5a8e504cbe34132b9: No bootstrap required, opened a new log
I20240501 07:59:00.679919 8052 ts_tablet_manager.cc:1367] T f9df6f3df2b94e5eb97dbff04dbe2ae1 P 1f9cb72be98941b5a8e504cbe34132b9: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20240501 07:59:00.680243 8051 raft_consensus.cc:729] T f9df6f3df2b94e5eb97dbff04dbe2ae1 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b8b3ad3aa8f4053b2167dd550f2086f, State: Initialized, Role: FOLLOWER
I20240501 07:59:00.680275 8053 tablet_bootstrap.cc:654] T 187940ba79f546b1ad79e7ec1de76442 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:00.680380 8052 raft_consensus.cc:348] T f9df6f3df2b94e5eb97dbff04dbe2ae1 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:59:00.680538 8052 raft_consensus.cc:374] T f9df6f3df2b94e5eb97dbff04dbe2ae1 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:00.680502 8051 consensus_queue.cc:260] T f9df6f3df2b94e5eb97dbff04dbe2ae1 P 8b8b3ad3aa8f4053b2167dd550f2086f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:59:00.680625 8052 raft_consensus.cc:729] T f9df6f3df2b94e5eb97dbff04dbe2ae1 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f9cb72be98941b5a8e504cbe34132b9, State: Initialized, Role: FOLLOWER
I20240501 07:59:00.680784 8052 consensus_queue.cc:260] T f9df6f3df2b94e5eb97dbff04dbe2ae1 P 1f9cb72be98941b5a8e504cbe34132b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:59:00.680874 8051 ts_tablet_manager.cc:1398] T f9df6f3df2b94e5eb97dbff04dbe2ae1 P 8b8b3ad3aa8f4053b2167dd550f2086f: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240501 07:59:00.681178 8052 ts_tablet_manager.cc:1398] T f9df6f3df2b94e5eb97dbff04dbe2ae1 P 1f9cb72be98941b5a8e504cbe34132b9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:00.681432 8052 tablet_bootstrap.cc:492] T 5122bc9a3cfb4b04aafc739256914bc2 P 1f9cb72be98941b5a8e504cbe34132b9: Bootstrap starting.
I20240501 07:59:00.681990 8050 tablet_bootstrap.cc:492] T f9df6f3df2b94e5eb97dbff04dbe2ae1 P 8315d22c56f7439a9e1fc82f153f018c: No bootstrap required, opened a new log
I20240501 07:59:00.682093 8050 ts_tablet_manager.cc:1367] T f9df6f3df2b94e5eb97dbff04dbe2ae1 P 8315d22c56f7439a9e1fc82f153f018c: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240501 07:59:00.682426 8052 tablet_bootstrap.cc:654] T 5122bc9a3cfb4b04aafc739256914bc2 P 1f9cb72be98941b5a8e504cbe34132b9: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:00.682548 8050 raft_consensus.cc:348] T f9df6f3df2b94e5eb97dbff04dbe2ae1 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:59:00.682682 8050 raft_consensus.cc:374] T f9df6f3df2b94e5eb97dbff04dbe2ae1 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:00.682718 8050 raft_consensus.cc:729] T f9df6f3df2b94e5eb97dbff04dbe2ae1 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8315d22c56f7439a9e1fc82f153f018c, State: Initialized, Role: FOLLOWER
I20240501 07:59:00.682871 8050 consensus_queue.cc:260] T f9df6f3df2b94e5eb97dbff04dbe2ae1 P 8315d22c56f7439a9e1fc82f153f018c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:59:00.682971 8053 tablet_bootstrap.cc:492] T 187940ba79f546b1ad79e7ec1de76442 P 3b164cbbc00f4e8eb3254ddd349a3ff9: No bootstrap required, opened a new log
I20240501 07:59:00.683056 8053 ts_tablet_manager.cc:1367] T 187940ba79f546b1ad79e7ec1de76442 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 07:59:00.683547 8053 raft_consensus.cc:348] T 187940ba79f546b1ad79e7ec1de76442 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:59:00.683624 8051 tablet_bootstrap.cc:492] T 187940ba79f546b1ad79e7ec1de76442 P 8b8b3ad3aa8f4053b2167dd550f2086f: Bootstrap starting.
I20240501 07:59:00.683655 8053 raft_consensus.cc:374] T 187940ba79f546b1ad79e7ec1de76442 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:00.683689 8053 raft_consensus.cc:729] T 187940ba79f546b1ad79e7ec1de76442 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b164cbbc00f4e8eb3254ddd349a3ff9, State: Initialized, Role: FOLLOWER
I20240501 07:59:00.683737 8052 tablet_bootstrap.cc:492] T 5122bc9a3cfb4b04aafc739256914bc2 P 1f9cb72be98941b5a8e504cbe34132b9: No bootstrap required, opened a new log
I20240501 07:59:00.683802 8052 ts_tablet_manager.cc:1367] T 5122bc9a3cfb4b04aafc739256914bc2 P 1f9cb72be98941b5a8e504cbe34132b9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240501 07:59:00.683804 8053 consensus_queue.cc:260] T 187940ba79f546b1ad79e7ec1de76442 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:59:00.684069 8053 ts_tablet_manager.cc:1398] T 187940ba79f546b1ad79e7ec1de76442 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:00.684209 8052 raft_consensus.cc:348] T 5122bc9a3cfb4b04aafc739256914bc2 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } }
I20240501 07:59:00.684305 8052 raft_consensus.cc:374] T 5122bc9a3cfb4b04aafc739256914bc2 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:00.684335 8052 raft_consensus.cc:729] T 5122bc9a3cfb4b04aafc739256914bc2 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f9cb72be98941b5a8e504cbe34132b9, State: Initialized, Role: FOLLOWER
I20240501 07:59:00.684434 8052 consensus_queue.cc:260] T 5122bc9a3cfb4b04aafc739256914bc2 P 1f9cb72be98941b5a8e504cbe34132b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } }
I20240501 07:59:00.684468 8051 tablet_bootstrap.cc:654] T 187940ba79f546b1ad79e7ec1de76442 P 8b8b3ad3aa8f4053b2167dd550f2086f: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:00.684670 8052 ts_tablet_manager.cc:1398] T 5122bc9a3cfb4b04aafc739256914bc2 P 1f9cb72be98941b5a8e504cbe34132b9: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:59:00.684815 8052 tablet_bootstrap.cc:492] T 187940ba79f546b1ad79e7ec1de76442 P 1f9cb72be98941b5a8e504cbe34132b9: Bootstrap starting.
I20240501 07:59:00.685912 8052 tablet_bootstrap.cc:654] T 187940ba79f546b1ad79e7ec1de76442 P 1f9cb72be98941b5a8e504cbe34132b9: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:00.686285 8050 ts_tablet_manager.cc:1398] T f9df6f3df2b94e5eb97dbff04dbe2ae1 P 8315d22c56f7439a9e1fc82f153f018c: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240501 07:59:00.686442 8050 tablet_bootstrap.cc:492] T 5122bc9a3cfb4b04aafc739256914bc2 P 8315d22c56f7439a9e1fc82f153f018c: Bootstrap starting.
I20240501 07:59:00.687052 8052 tablet_bootstrap.cc:492] T 187940ba79f546b1ad79e7ec1de76442 P 1f9cb72be98941b5a8e504cbe34132b9: No bootstrap required, opened a new log
I20240501 07:59:00.687137 8052 ts_tablet_manager.cc:1367] T 187940ba79f546b1ad79e7ec1de76442 P 1f9cb72be98941b5a8e504cbe34132b9: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 07:59:00.687345 8050 tablet_bootstrap.cc:654] T 5122bc9a3cfb4b04aafc739256914bc2 P 8315d22c56f7439a9e1fc82f153f018c: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:00.687543 8052 raft_consensus.cc:348] T 187940ba79f546b1ad79e7ec1de76442 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:59:00.687629 8052 raft_consensus.cc:374] T 187940ba79f546b1ad79e7ec1de76442 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:00.687659 8052 raft_consensus.cc:729] T 187940ba79f546b1ad79e7ec1de76442 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f9cb72be98941b5a8e504cbe34132b9, State: Initialized, Role: FOLLOWER
I20240501 07:59:00.687762 8052 consensus_queue.cc:260] T 187940ba79f546b1ad79e7ec1de76442 P 1f9cb72be98941b5a8e504cbe34132b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:59:00.688015 8052 ts_tablet_manager.cc:1398] T 187940ba79f546b1ad79e7ec1de76442 P 1f9cb72be98941b5a8e504cbe34132b9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:00.688947 8051 tablet_bootstrap.cc:492] T 187940ba79f546b1ad79e7ec1de76442 P 8b8b3ad3aa8f4053b2167dd550f2086f: No bootstrap required, opened a new log
I20240501 07:59:00.689116 8051 ts_tablet_manager.cc:1367] T 187940ba79f546b1ad79e7ec1de76442 P 8b8b3ad3aa8f4053b2167dd550f2086f: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.002s
I20240501 07:59:00.689605 8051 raft_consensus.cc:348] T 187940ba79f546b1ad79e7ec1de76442 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:59:00.689749 8051 raft_consensus.cc:374] T 187940ba79f546b1ad79e7ec1de76442 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:00.689817 8051 raft_consensus.cc:729] T 187940ba79f546b1ad79e7ec1de76442 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b8b3ad3aa8f4053b2167dd550f2086f, State: Initialized, Role: FOLLOWER
I20240501 07:59:00.690040 8050 tablet_bootstrap.cc:492] T 5122bc9a3cfb4b04aafc739256914bc2 P 8315d22c56f7439a9e1fc82f153f018c: No bootstrap required, opened a new log
I20240501 07:59:00.690018 8051 consensus_queue.cc:260] T 187940ba79f546b1ad79e7ec1de76442 P 8b8b3ad3aa8f4053b2167dd550f2086f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:59:00.690120 8050 ts_tablet_manager.cc:1367] T 5122bc9a3cfb4b04aafc739256914bc2 P 8315d22c56f7439a9e1fc82f153f018c: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 07:59:00.690384 8051 ts_tablet_manager.cc:1398] T 187940ba79f546b1ad79e7ec1de76442 P 8b8b3ad3aa8f4053b2167dd550f2086f: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 07:59:00.690591 8051 tablet_bootstrap.cc:492] T 5122bc9a3cfb4b04aafc739256914bc2 P 8b8b3ad3aa8f4053b2167dd550f2086f: Bootstrap starting.
I20240501 07:59:00.691200 8050 raft_consensus.cc:348] T 5122bc9a3cfb4b04aafc739256914bc2 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } }
I20240501 07:59:00.691303 8050 raft_consensus.cc:374] T 5122bc9a3cfb4b04aafc739256914bc2 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:00.691344 8050 raft_consensus.cc:729] T 5122bc9a3cfb4b04aafc739256914bc2 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8315d22c56f7439a9e1fc82f153f018c, State: Initialized, Role: FOLLOWER
I20240501 07:59:00.691454 8050 consensus_queue.cc:260] T 5122bc9a3cfb4b04aafc739256914bc2 P 8315d22c56f7439a9e1fc82f153f018c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } }
I20240501 07:59:00.691650 8051 tablet_bootstrap.cc:654] T 5122bc9a3cfb4b04aafc739256914bc2 P 8b8b3ad3aa8f4053b2167dd550f2086f: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:00.691771 8050 ts_tablet_manager.cc:1398] T 5122bc9a3cfb4b04aafc739256914bc2 P 8315d22c56f7439a9e1fc82f153f018c: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:59:00.692685 8051 tablet_bootstrap.cc:492] T 5122bc9a3cfb4b04aafc739256914bc2 P 8b8b3ad3aa8f4053b2167dd550f2086f: No bootstrap required, opened a new log
I20240501 07:59:00.692816 8051 ts_tablet_manager.cc:1367] T 5122bc9a3cfb4b04aafc739256914bc2 P 8b8b3ad3aa8f4053b2167dd550f2086f: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240501 07:59:00.693413 8051 raft_consensus.cc:348] T 5122bc9a3cfb4b04aafc739256914bc2 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } }
I20240501 07:59:00.693516 8051 raft_consensus.cc:374] T 5122bc9a3cfb4b04aafc739256914bc2 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:00.693552 8051 raft_consensus.cc:729] T 5122bc9a3cfb4b04aafc739256914bc2 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b8b3ad3aa8f4053b2167dd550f2086f, State: Initialized, Role: FOLLOWER
I20240501 07:59:00.693678 8051 consensus_queue.cc:260] T 5122bc9a3cfb4b04aafc739256914bc2 P 8b8b3ad3aa8f4053b2167dd550f2086f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } }
I20240501 07:59:00.694026 8051 ts_tablet_manager.cc:1398] T 5122bc9a3cfb4b04aafc739256914bc2 P 8b8b3ad3aa8f4053b2167dd550f2086f: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 07:59:00.741972 8035 raft_consensus.cc:482] T 187940ba79f546b1ad79e7ec1de76442 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:00.742133 8035 raft_consensus.cc:504] T 187940ba79f546b1ad79e7ec1de76442 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:59:00.742580 8035 leader_election.cc:288] T 187940ba79f546b1ad79e7ec1de76442 P 1f9cb72be98941b5a8e504cbe34132b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129:46875), 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131:34471)
I20240501 07:59:00.742997 6575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "187940ba79f546b1ad79e7ec1de76442" candidate_uuid: "1f9cb72be98941b5a8e504cbe34132b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" is_pre_election: true
I20240501 07:59:00.743052 6314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "187940ba79f546b1ad79e7ec1de76442" candidate_uuid: "1f9cb72be98941b5a8e504cbe34132b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" is_pre_election: true
I20240501 07:59:00.743139 6575 raft_consensus.cc:2454] T 187940ba79f546b1ad79e7ec1de76442 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f9cb72be98941b5a8e504cbe34132b9 in term 0.
I20240501 07:59:00.743235 6314 raft_consensus.cc:2454] T 187940ba79f546b1ad79e7ec1de76442 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f9cb72be98941b5a8e504cbe34132b9 in term 0.
I20240501 07:59:00.743419 6644 leader_election.cc:302] T 187940ba79f546b1ad79e7ec1de76442 P 1f9cb72be98941b5a8e504cbe34132b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f9cb72be98941b5a8e504cbe34132b9, 3b164cbbc00f4e8eb3254ddd349a3ff9; no voters:
I20240501 07:59:00.743609 8035 raft_consensus.cc:2781] T 187940ba79f546b1ad79e7ec1de76442 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:00.743671 8035 raft_consensus.cc:482] T 187940ba79f546b1ad79e7ec1de76442 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:00.743705 8035 raft_consensus.cc:3037] T 187940ba79f546b1ad79e7ec1de76442 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:00.744560 8035 raft_consensus.cc:504] T 187940ba79f546b1ad79e7ec1de76442 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:59:00.744871 8035 leader_election.cc:288] T 187940ba79f546b1ad79e7ec1de76442 P 1f9cb72be98941b5a8e504cbe34132b9 [CANDIDATE]: Term 1 election: Requested vote from peers 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129:46875), 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131:34471)
I20240501 07:59:00.745260 6314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "187940ba79f546b1ad79e7ec1de76442" candidate_uuid: "1f9cb72be98941b5a8e504cbe34132b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f"
I20240501 07:59:00.745282 6575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "187940ba79f546b1ad79e7ec1de76442" candidate_uuid: "1f9cb72be98941b5a8e504cbe34132b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9"
I20240501 07:59:00.745407 6575 raft_consensus.cc:3037] T 187940ba79f546b1ad79e7ec1de76442 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:00.745407 6314 raft_consensus.cc:3037] T 187940ba79f546b1ad79e7ec1de76442 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:00.746129 6575 raft_consensus.cc:2454] T 187940ba79f546b1ad79e7ec1de76442 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f9cb72be98941b5a8e504cbe34132b9 in term 1.
I20240501 07:59:00.746398 6644 leader_election.cc:302] T 187940ba79f546b1ad79e7ec1de76442 P 1f9cb72be98941b5a8e504cbe34132b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f9cb72be98941b5a8e504cbe34132b9, 3b164cbbc00f4e8eb3254ddd349a3ff9; no voters:
I20240501 07:59:00.746537 8035 raft_consensus.cc:2781] T 187940ba79f546b1ad79e7ec1de76442 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:00.746601 8035 raft_consensus.cc:686] T 187940ba79f546b1ad79e7ec1de76442 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 LEADER]: Becoming Leader. State: Replica: 1f9cb72be98941b5a8e504cbe34132b9, State: Running, Role: LEADER
I20240501 07:59:00.746655 6314 raft_consensus.cc:2454] T 187940ba79f546b1ad79e7ec1de76442 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f9cb72be98941b5a8e504cbe34132b9 in term 1.
I20240501 07:59:00.746737 8035 consensus_queue.cc:237] T 187940ba79f546b1ad79e7ec1de76442 P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:59:00.749709 6172 catalog_manager.cc:5484] T 187940ba79f546b1ad79e7ec1de76442 P 1f9cb72be98941b5a8e504cbe34132b9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132). New cstate: current_term: 1 leader_uuid: "1f9cb72be98941b5a8e504cbe34132b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } health_report { overall_health: HEALTHY } } }
I20240501 07:59:00.799558 7306 raft_consensus.cc:482] T 5122bc9a3cfb4b04aafc739256914bc2 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:00.799681 7306 raft_consensus.cc:504] T 5122bc9a3cfb4b04aafc739256914bc2 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } }
I20240501 07:59:00.800084 7306 leader_election.cc:288] T 5122bc9a3cfb4b04aafc739256914bc2 P 8b8b3ad3aa8f4053b2167dd550f2086f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132:36773), 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130:32941)
I20240501 07:59:00.800534 6708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5122bc9a3cfb4b04aafc739256914bc2" candidate_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f9cb72be98941b5a8e504cbe34132b9" is_pre_election: true
I20240501 07:59:00.800534 6445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5122bc9a3cfb4b04aafc739256914bc2" candidate_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8315d22c56f7439a9e1fc82f153f018c" is_pre_election: true
I20240501 07:59:00.800680 6445 raft_consensus.cc:2454] T 5122bc9a3cfb4b04aafc739256914bc2 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b8b3ad3aa8f4053b2167dd550f2086f in term 0.
I20240501 07:59:00.800679 6708 raft_consensus.cc:2454] T 5122bc9a3cfb4b04aafc739256914bc2 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b8b3ad3aa8f4053b2167dd550f2086f in term 0.
I20240501 07:59:00.801004 6250 leader_election.cc:302] T 5122bc9a3cfb4b04aafc739256914bc2 P 8b8b3ad3aa8f4053b2167dd550f2086f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f9cb72be98941b5a8e504cbe34132b9, 8b8b3ad3aa8f4053b2167dd550f2086f; no voters:
I20240501 07:59:00.801174 7306 raft_consensus.cc:2781] T 5122bc9a3cfb4b04aafc739256914bc2 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:00.801232 7306 raft_consensus.cc:482] T 5122bc9a3cfb4b04aafc739256914bc2 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:00.801280 7306 raft_consensus.cc:3037] T 5122bc9a3cfb4b04aafc739256914bc2 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:00.802086 7306 raft_consensus.cc:504] T 5122bc9a3cfb4b04aafc739256914bc2 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } }
I20240501 07:59:00.802405 7306 leader_election.cc:288] T 5122bc9a3cfb4b04aafc739256914bc2 P 8b8b3ad3aa8f4053b2167dd550f2086f [CANDIDATE]: Term 1 election: Requested vote from peers 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132:36773), 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130:32941)
I20240501 07:59:00.802752 6445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5122bc9a3cfb4b04aafc739256914bc2" candidate_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8315d22c56f7439a9e1fc82f153f018c"
I20240501 07:59:00.802745 6708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5122bc9a3cfb4b04aafc739256914bc2" candidate_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f9cb72be98941b5a8e504cbe34132b9"
I20240501 07:59:00.802871 6445 raft_consensus.cc:3037] T 5122bc9a3cfb4b04aafc739256914bc2 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:00.802871 6708 raft_consensus.cc:3037] T 5122bc9a3cfb4b04aafc739256914bc2 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:00.803838 6445 raft_consensus.cc:2454] T 5122bc9a3cfb4b04aafc739256914bc2 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b8b3ad3aa8f4053b2167dd550f2086f in term 1.
I20240501 07:59:00.803838 6708 raft_consensus.cc:2454] T 5122bc9a3cfb4b04aafc739256914bc2 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b8b3ad3aa8f4053b2167dd550f2086f in term 1.
I20240501 07:59:00.804157 6250 leader_election.cc:302] T 5122bc9a3cfb4b04aafc739256914bc2 P 8b8b3ad3aa8f4053b2167dd550f2086f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f9cb72be98941b5a8e504cbe34132b9, 8b8b3ad3aa8f4053b2167dd550f2086f; no voters:
I20240501 07:59:00.804359 7306 raft_consensus.cc:2781] T 5122bc9a3cfb4b04aafc739256914bc2 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:00.804459 7306 raft_consensus.cc:686] T 5122bc9a3cfb4b04aafc739256914bc2 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 LEADER]: Becoming Leader. State: Replica: 8b8b3ad3aa8f4053b2167dd550f2086f, State: Running, Role: LEADER
I20240501 07:59:00.804616 7306 consensus_queue.cc:237] T 5122bc9a3cfb4b04aafc739256914bc2 P 8b8b3ad3aa8f4053b2167dd550f2086f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } }
I20240501 07:59:00.806452 6172 catalog_manager.cc:5484] T 5122bc9a3cfb4b04aafc739256914bc2 P 8b8b3ad3aa8f4053b2167dd550f2086f reported cstate change: term changed from 0 to 1, leader changed from <none> to 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129). New cstate: current_term: 1 leader_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } health_report { overall_health: UNKNOWN } } }
I20240501 07:59:00.807400 8029 raft_consensus.cc:482] T f9df6f3df2b94e5eb97dbff04dbe2ae1 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:00.807494 8029 raft_consensus.cc:504] T f9df6f3df2b94e5eb97dbff04dbe2ae1 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:59:00.807893 8029 leader_election.cc:288] T f9df6f3df2b94e5eb97dbff04dbe2ae1 P 8315d22c56f7439a9e1fc82f153f018c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129:46875), 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132:36773)
I20240501 07:59:00.808432 6314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9df6f3df2b94e5eb97dbff04dbe2ae1" candidate_uuid: "8315d22c56f7439a9e1fc82f153f018c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" is_pre_election: true
I20240501 07:59:00.808552 6314 raft_consensus.cc:2454] T f9df6f3df2b94e5eb97dbff04dbe2ae1 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8315d22c56f7439a9e1fc82f153f018c in term 0.
I20240501 07:59:00.808542 6708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9df6f3df2b94e5eb97dbff04dbe2ae1" candidate_uuid: "8315d22c56f7439a9e1fc82f153f018c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f9cb72be98941b5a8e504cbe34132b9" is_pre_election: true
I20240501 07:59:00.808647 6708 raft_consensus.cc:2454] T f9df6f3df2b94e5eb97dbff04dbe2ae1 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8315d22c56f7439a9e1fc82f153f018c in term 0.
I20240501 07:59:00.808885 6379 leader_election.cc:302] T f9df6f3df2b94e5eb97dbff04dbe2ae1 P 8315d22c56f7439a9e1fc82f153f018c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8315d22c56f7439a9e1fc82f153f018c, 8b8b3ad3aa8f4053b2167dd550f2086f; no voters:
I20240501 07:59:00.809059 8029 raft_consensus.cc:2781] T f9df6f3df2b94e5eb97dbff04dbe2ae1 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:00.809128 8029 raft_consensus.cc:482] T f9df6f3df2b94e5eb97dbff04dbe2ae1 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:00.809172 8029 raft_consensus.cc:3037] T f9df6f3df2b94e5eb97dbff04dbe2ae1 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:00.810066 8029 raft_consensus.cc:504] T f9df6f3df2b94e5eb97dbff04dbe2ae1 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:59:00.810366 8029 leader_election.cc:288] T f9df6f3df2b94e5eb97dbff04dbe2ae1 P 8315d22c56f7439a9e1fc82f153f018c [CANDIDATE]: Term 1 election: Requested vote from peers 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129:46875), 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132:36773)
I20240501 07:59:00.810642 6314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9df6f3df2b94e5eb97dbff04dbe2ae1" candidate_uuid: "8315d22c56f7439a9e1fc82f153f018c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f"
I20240501 07:59:00.810750 6314 raft_consensus.cc:3037] T f9df6f3df2b94e5eb97dbff04dbe2ae1 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:00.810752 6708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9df6f3df2b94e5eb97dbff04dbe2ae1" candidate_uuid: "8315d22c56f7439a9e1fc82f153f018c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f9cb72be98941b5a8e504cbe34132b9"
I20240501 07:59:00.810860 6708 raft_consensus.cc:3037] T f9df6f3df2b94e5eb97dbff04dbe2ae1 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:00.811690 6708 raft_consensus.cc:2454] T f9df6f3df2b94e5eb97dbff04dbe2ae1 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8315d22c56f7439a9e1fc82f153f018c in term 1.
I20240501 07:59:00.811690 6314 raft_consensus.cc:2454] T f9df6f3df2b94e5eb97dbff04dbe2ae1 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8315d22c56f7439a9e1fc82f153f018c in term 1.
I20240501 07:59:00.811960 6381 leader_election.cc:302] T f9df6f3df2b94e5eb97dbff04dbe2ae1 P 8315d22c56f7439a9e1fc82f153f018c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f9cb72be98941b5a8e504cbe34132b9, 8315d22c56f7439a9e1fc82f153f018c; no voters:
I20240501 07:59:00.812181 8029 raft_consensus.cc:2781] T f9df6f3df2b94e5eb97dbff04dbe2ae1 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:00.812283 8029 raft_consensus.cc:686] T f9df6f3df2b94e5eb97dbff04dbe2ae1 P 8315d22c56f7439a9e1fc82f153f018c [term 1 LEADER]: Becoming Leader. State: Replica: 8315d22c56f7439a9e1fc82f153f018c, State: Running, Role: LEADER
I20240501 07:59:00.812433 8029 consensus_queue.cc:237] T f9df6f3df2b94e5eb97dbff04dbe2ae1 P 8315d22c56f7439a9e1fc82f153f018c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:59:00.814230 6172 catalog_manager.cc:5484] T f9df6f3df2b94e5eb97dbff04dbe2ae1 P 8315d22c56f7439a9e1fc82f153f018c reported cstate change: term changed from 0 to 1, leader changed from <none> to 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130). New cstate: current_term: 1 leader_uuid: "8315d22c56f7439a9e1fc82f153f018c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } health_report { overall_health: UNKNOWN } } }
W20240501 07:59:00.952831 8066 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:59:00.953197 8066 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:59:00.956463 8066 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_10'
I20240501 07:59:01.049746 6708 raft_consensus.cc:1261] T f9df6f3df2b94e5eb97dbff04dbe2ae1 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Refusing update from remote peer 8315d22c56f7439a9e1fc82f153f018c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:01.050704 8029 consensus_queue.cc:1035] T f9df6f3df2b94e5eb97dbff04dbe2ae1 P 8315d22c56f7439a9e1fc82f153f018c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:01.051357 6314 raft_consensus.cc:1261] T 187940ba79f546b1ad79e7ec1de76442 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Refusing update from remote peer 1f9cb72be98941b5a8e504cbe34132b9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:01.051399 6577 raft_consensus.cc:1261] T 187940ba79f546b1ad79e7ec1de76442 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Refusing update from remote peer 1f9cb72be98941b5a8e504cbe34132b9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:01.052219 7959 consensus_queue.cc:1035] T 187940ba79f546b1ad79e7ec1de76442 P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:01.052371 7963 consensus_queue.cc:1035] T 187940ba79f546b1ad79e7ec1de76442 P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:01.053112 6707 raft_consensus.cc:1261] T 5122bc9a3cfb4b04aafc739256914bc2 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Refusing update from remote peer 8b8b3ad3aa8f4053b2167dd550f2086f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:01.053223 6445 raft_consensus.cc:1261] T 5122bc9a3cfb4b04aafc739256914bc2 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Refusing update from remote peer 8b8b3ad3aa8f4053b2167dd550f2086f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:01.056854 7306 consensus_queue.cc:1035] T 5122bc9a3cfb4b04aafc739256914bc2 P 8b8b3ad3aa8f4053b2167dd550f2086f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:01.057152 7306 consensus_queue.cc:1035] T 5122bc9a3cfb4b04aafc739256914bc2 P 8b8b3ad3aa8f4053b2167dd550f2086f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:01.058167 6313 raft_consensus.cc:1261] T f9df6f3df2b94e5eb97dbff04dbe2ae1 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Refusing update from remote peer 8315d22c56f7439a9e1fc82f153f018c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:01.059221 8045 consensus_queue.cc:1035] T f9df6f3df2b94e5eb97dbff04dbe2ae1 P 8315d22c56f7439a9e1fc82f153f018c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 127.543 ms
time per row: 0.0637714 ms
I20240501 07:59:01.127369 6172 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55950:
name: "rebalance_test_table_11"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 07:59:01.132483 6293 tablet_service.cc:1467] Processing CreateTablet for tablet d6c28cd91f624673b8e428f471a1dcce (DEFAULT_TABLE table=rebalance_test_table_11 [id=3e062fd8d58941efbd3d190c28bb0059]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:01.132848 6293 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6c28cd91f624673b8e428f471a1dcce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:01.132844 6557 tablet_service.cc:1467] Processing CreateTablet for tablet d6c28cd91f624673b8e428f471a1dcce (DEFAULT_TABLE table=rebalance_test_table_11 [id=3e062fd8d58941efbd3d190c28bb0059]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:01.133189 6557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6c28cd91f624673b8e428f471a1dcce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:01.133553 6687 tablet_service.cc:1467] Processing CreateTablet for tablet b7d57b7fd5234f50afbc859bc997f3e1 (DEFAULT_TABLE table=rebalance_test_table_11 [id=3e062fd8d58941efbd3d190c28bb0059]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:01.133653 6686 tablet_service.cc:1467] Processing CreateTablet for tablet a4727883b2544ed58b356f41dc8017a9 (DEFAULT_TABLE table=rebalance_test_table_11 [id=3e062fd8d58941efbd3d190c28bb0059]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:01.133809 6687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7d57b7fd5234f50afbc859bc997f3e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:01.134199 6686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4727883b2544ed58b356f41dc8017a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:01.135303 6426 tablet_service.cc:1467] Processing CreateTablet for tablet d6c28cd91f624673b8e428f471a1dcce (DEFAULT_TABLE table=rebalance_test_table_11 [id=3e062fd8d58941efbd3d190c28bb0059]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:01.135370 8051 tablet_bootstrap.cc:492] T d6c28cd91f624673b8e428f471a1dcce P 8b8b3ad3aa8f4053b2167dd550f2086f: Bootstrap starting.
I20240501 07:59:01.135596 6426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6c28cd91f624673b8e428f471a1dcce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:01.136312 8051 tablet_bootstrap.cc:654] T d6c28cd91f624673b8e428f471a1dcce P 8b8b3ad3aa8f4053b2167dd550f2086f: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:01.136456 8053 tablet_bootstrap.cc:492] T d6c28cd91f624673b8e428f471a1dcce P 3b164cbbc00f4e8eb3254ddd349a3ff9: Bootstrap starting.
I20240501 07:59:01.137369 8053 tablet_bootstrap.cc:654] T d6c28cd91f624673b8e428f471a1dcce P 3b164cbbc00f4e8eb3254ddd349a3ff9: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:01.137485 8052 tablet_bootstrap.cc:492] T b7d57b7fd5234f50afbc859bc997f3e1 P 1f9cb72be98941b5a8e504cbe34132b9: Bootstrap starting.
I20240501 07:59:01.137486 8051 tablet_bootstrap.cc:492] T d6c28cd91f624673b8e428f471a1dcce P 8b8b3ad3aa8f4053b2167dd550f2086f: No bootstrap required, opened a new log
I20240501 07:59:01.137566 8051 ts_tablet_manager.cc:1367] T d6c28cd91f624673b8e428f471a1dcce P 8b8b3ad3aa8f4053b2167dd550f2086f: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240501 07:59:01.137984 8051 raft_consensus.cc:348] T d6c28cd91f624673b8e428f471a1dcce P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } }
I20240501 07:59:01.138002 6555 tablet_service.cc:1467] Processing CreateTablet for tablet a4727883b2544ed58b356f41dc8017a9 (DEFAULT_TABLE table=rebalance_test_table_11 [id=3e062fd8d58941efbd3d190c28bb0059]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:01.138096 8051 raft_consensus.cc:374] T d6c28cd91f624673b8e428f471a1dcce P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:01.138130 8051 raft_consensus.cc:729] T d6c28cd91f624673b8e428f471a1dcce P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b8b3ad3aa8f4053b2167dd550f2086f, State: Initialized, Role: FOLLOWER
I20240501 07:59:01.138231 6555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4727883b2544ed58b356f41dc8017a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:01.138262 8052 tablet_bootstrap.cc:654] T b7d57b7fd5234f50afbc859bc997f3e1 P 1f9cb72be98941b5a8e504cbe34132b9: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:01.138269 8051 consensus_queue.cc:260] T d6c28cd91f624673b8e428f471a1dcce P 8b8b3ad3aa8f4053b2167dd550f2086f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } }
I20240501 07:59:01.138571 8051 ts_tablet_manager.cc:1398] T d6c28cd91f624673b8e428f471a1dcce P 8b8b3ad3aa8f4053b2167dd550f2086f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:59:01.138784 8050 tablet_bootstrap.cc:492] T d6c28cd91f624673b8e428f471a1dcce P 8315d22c56f7439a9e1fc82f153f018c: Bootstrap starting.
I20240501 07:59:01.138937 6557 tablet_service.cc:1467] Processing CreateTablet for tablet b7d57b7fd5234f50afbc859bc997f3e1 (DEFAULT_TABLE table=rebalance_test_table_11 [id=3e062fd8d58941efbd3d190c28bb0059]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:01.139142 6557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7d57b7fd5234f50afbc859bc997f3e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:01.139489 6426 tablet_service.cc:1467] Processing CreateTablet for tablet a4727883b2544ed58b356f41dc8017a9 (DEFAULT_TABLE table=rebalance_test_table_11 [id=3e062fd8d58941efbd3d190c28bb0059]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:01.139607 8050 tablet_bootstrap.cc:654] T d6c28cd91f624673b8e428f471a1dcce P 8315d22c56f7439a9e1fc82f153f018c: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:01.139710 6426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4727883b2544ed58b356f41dc8017a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:01.141166 6293 tablet_service.cc:1467] Processing CreateTablet for tablet b7d57b7fd5234f50afbc859bc997f3e1 (DEFAULT_TABLE table=rebalance_test_table_11 [id=3e062fd8d58941efbd3d190c28bb0059]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:01.141397 6293 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7d57b7fd5234f50afbc859bc997f3e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:01.141902 8053 tablet_bootstrap.cc:492] T d6c28cd91f624673b8e428f471a1dcce P 3b164cbbc00f4e8eb3254ddd349a3ff9: No bootstrap required, opened a new log
I20240501 07:59:01.141984 8053 ts_tablet_manager.cc:1367] T d6c28cd91f624673b8e428f471a1dcce P 3b164cbbc00f4e8eb3254ddd349a3ff9: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 07:59:01.142424 8053 raft_consensus.cc:348] T d6c28cd91f624673b8e428f471a1dcce P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } }
I20240501 07:59:01.142524 8053 raft_consensus.cc:374] T d6c28cd91f624673b8e428f471a1dcce P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:01.142566 8053 raft_consensus.cc:729] T d6c28cd91f624673b8e428f471a1dcce P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b164cbbc00f4e8eb3254ddd349a3ff9, State: Initialized, Role: FOLLOWER
I20240501 07:59:01.142694 8053 consensus_queue.cc:260] T d6c28cd91f624673b8e428f471a1dcce P 3b164cbbc00f4e8eb3254ddd349a3ff9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } }
I20240501 07:59:01.142999 8053 ts_tablet_manager.cc:1398] T d6c28cd91f624673b8e428f471a1dcce P 3b164cbbc00f4e8eb3254ddd349a3ff9: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:59:01.143110 8052 tablet_bootstrap.cc:492] T b7d57b7fd5234f50afbc859bc997f3e1 P 1f9cb72be98941b5a8e504cbe34132b9: No bootstrap required, opened a new log
I20240501 07:59:01.143146 8053 tablet_bootstrap.cc:492] T a4727883b2544ed58b356f41dc8017a9 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Bootstrap starting.
I20240501 07:59:01.143193 8052 ts_tablet_manager.cc:1367] T b7d57b7fd5234f50afbc859bc997f3e1 P 1f9cb72be98941b5a8e504cbe34132b9: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.001s
I20240501 07:59:01.143656 8052 raft_consensus.cc:348] T b7d57b7fd5234f50afbc859bc997f3e1 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:59:01.143745 8052 raft_consensus.cc:374] T b7d57b7fd5234f50afbc859bc997f3e1 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:01.143767 8052 raft_consensus.cc:729] T b7d57b7fd5234f50afbc859bc997f3e1 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f9cb72be98941b5a8e504cbe34132b9, State: Initialized, Role: FOLLOWER
I20240501 07:59:01.143882 8052 consensus_queue.cc:260] T b7d57b7fd5234f50afbc859bc997f3e1 P 1f9cb72be98941b5a8e504cbe34132b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:59:01.143994 8053 tablet_bootstrap.cc:654] T a4727883b2544ed58b356f41dc8017a9 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:01.144156 8052 ts_tablet_manager.cc:1398] T b7d57b7fd5234f50afbc859bc997f3e1 P 1f9cb72be98941b5a8e504cbe34132b9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:01.144322 8052 tablet_bootstrap.cc:492] T a4727883b2544ed58b356f41dc8017a9 P 1f9cb72be98941b5a8e504cbe34132b9: Bootstrap starting.
I20240501 07:59:01.144389 8050 tablet_bootstrap.cc:492] T d6c28cd91f624673b8e428f471a1dcce P 8315d22c56f7439a9e1fc82f153f018c: No bootstrap required, opened a new log
I20240501 07:59:01.144451 8050 ts_tablet_manager.cc:1367] T d6c28cd91f624673b8e428f471a1dcce P 8315d22c56f7439a9e1fc82f153f018c: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20240501 07:59:01.144999 8053 tablet_bootstrap.cc:492] T a4727883b2544ed58b356f41dc8017a9 P 3b164cbbc00f4e8eb3254ddd349a3ff9: No bootstrap required, opened a new log
I20240501 07:59:01.144881 8050 raft_consensus.cc:348] T d6c28cd91f624673b8e428f471a1dcce P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } }
I20240501 07:59:01.145077 8053 ts_tablet_manager.cc:1367] T a4727883b2544ed58b356f41dc8017a9 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 07:59:01.145093 8050 raft_consensus.cc:374] T d6c28cd91f624673b8e428f471a1dcce P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:01.145141 8050 raft_consensus.cc:729] T d6c28cd91f624673b8e428f471a1dcce P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8315d22c56f7439a9e1fc82f153f018c, State: Initialized, Role: FOLLOWER
I20240501 07:59:01.145231 8052 tablet_bootstrap.cc:654] T a4727883b2544ed58b356f41dc8017a9 P 1f9cb72be98941b5a8e504cbe34132b9: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:01.145274 8050 consensus_queue.cc:260] T d6c28cd91f624673b8e428f471a1dcce P 8315d22c56f7439a9e1fc82f153f018c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } }
I20240501 07:59:01.145433 8053 raft_consensus.cc:348] T a4727883b2544ed58b356f41dc8017a9 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:59:01.145531 8053 raft_consensus.cc:374] T a4727883b2544ed58b356f41dc8017a9 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:01.145562 8053 raft_consensus.cc:729] T a4727883b2544ed58b356f41dc8017a9 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b164cbbc00f4e8eb3254ddd349a3ff9, State: Initialized, Role: FOLLOWER
I20240501 07:59:01.145568 8050 ts_tablet_manager.cc:1398] T d6c28cd91f624673b8e428f471a1dcce P 8315d22c56f7439a9e1fc82f153f018c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:01.145668 8053 consensus_queue.cc:260] T a4727883b2544ed58b356f41dc8017a9 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:59:01.145735 8050 tablet_bootstrap.cc:492] T a4727883b2544ed58b356f41dc8017a9 P 8315d22c56f7439a9e1fc82f153f018c: Bootstrap starting.
I20240501 07:59:01.145936 8053 ts_tablet_manager.cc:1398] T a4727883b2544ed58b356f41dc8017a9 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:01.146098 8053 tablet_bootstrap.cc:492] T b7d57b7fd5234f50afbc859bc997f3e1 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Bootstrap starting.
I20240501 07:59:01.146834 8053 tablet_bootstrap.cc:654] T b7d57b7fd5234f50afbc859bc997f3e1 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:01.148255 8052 tablet_bootstrap.cc:492] T a4727883b2544ed58b356f41dc8017a9 P 1f9cb72be98941b5a8e504cbe34132b9: No bootstrap required, opened a new log
I20240501 07:59:01.148357 8052 ts_tablet_manager.cc:1367] T a4727883b2544ed58b356f41dc8017a9 P 1f9cb72be98941b5a8e504cbe34132b9: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 07:59:01.148494 8051 tablet_bootstrap.cc:492] T b7d57b7fd5234f50afbc859bc997f3e1 P 8b8b3ad3aa8f4053b2167dd550f2086f: Bootstrap starting.
I20240501 07:59:01.148751 8052 raft_consensus.cc:348] T a4727883b2544ed58b356f41dc8017a9 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:59:01.148845 8052 raft_consensus.cc:374] T a4727883b2544ed58b356f41dc8017a9 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:01.148877 8052 raft_consensus.cc:729] T a4727883b2544ed58b356f41dc8017a9 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f9cb72be98941b5a8e504cbe34132b9, State: Initialized, Role: FOLLOWER
I20240501 07:59:01.149015 8052 consensus_queue.cc:260] T a4727883b2544ed58b356f41dc8017a9 P 1f9cb72be98941b5a8e504cbe34132b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:59:01.149117 8050 tablet_bootstrap.cc:654] T a4727883b2544ed58b356f41dc8017a9 P 8315d22c56f7439a9e1fc82f153f018c: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:01.149276 8052 ts_tablet_manager.cc:1398] T a4727883b2544ed58b356f41dc8017a9 P 1f9cb72be98941b5a8e504cbe34132b9: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 07:59:01.150290 8053 tablet_bootstrap.cc:492] T b7d57b7fd5234f50afbc859bc997f3e1 P 3b164cbbc00f4e8eb3254ddd349a3ff9: No bootstrap required, opened a new log
I20240501 07:59:01.150372 8053 ts_tablet_manager.cc:1367] T b7d57b7fd5234f50afbc859bc997f3e1 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240501 07:59:01.150851 8053 raft_consensus.cc:348] T b7d57b7fd5234f50afbc859bc997f3e1 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:59:01.150946 8053 raft_consensus.cc:374] T b7d57b7fd5234f50afbc859bc997f3e1 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:01.150990 8053 raft_consensus.cc:729] T b7d57b7fd5234f50afbc859bc997f3e1 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b164cbbc00f4e8eb3254ddd349a3ff9, State: Initialized, Role: FOLLOWER
I20240501 07:59:01.151095 8053 consensus_queue.cc:260] T b7d57b7fd5234f50afbc859bc997f3e1 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:59:01.151219 8051 tablet_bootstrap.cc:654] T b7d57b7fd5234f50afbc859bc997f3e1 P 8b8b3ad3aa8f4053b2167dd550f2086f: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:01.151487 8053 ts_tablet_manager.cc:1398] T b7d57b7fd5234f50afbc859bc997f3e1 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:01.152282 8050 tablet_bootstrap.cc:492] T a4727883b2544ed58b356f41dc8017a9 P 8315d22c56f7439a9e1fc82f153f018c: No bootstrap required, opened a new log
I20240501 07:59:01.152371 8050 ts_tablet_manager.cc:1367] T a4727883b2544ed58b356f41dc8017a9 P 8315d22c56f7439a9e1fc82f153f018c: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240501 07:59:01.152482 8051 tablet_bootstrap.cc:492] T b7d57b7fd5234f50afbc859bc997f3e1 P 8b8b3ad3aa8f4053b2167dd550f2086f: No bootstrap required, opened a new log
I20240501 07:59:01.152565 8051 ts_tablet_manager.cc:1367] T b7d57b7fd5234f50afbc859bc997f3e1 P 8b8b3ad3aa8f4053b2167dd550f2086f: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240501 07:59:01.152894 8050 raft_consensus.cc:348] T a4727883b2544ed58b356f41dc8017a9 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:59:01.152937 8051 raft_consensus.cc:348] T b7d57b7fd5234f50afbc859bc997f3e1 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:59:01.153038 8050 raft_consensus.cc:374] T a4727883b2544ed58b356f41dc8017a9 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:01.153052 8051 raft_consensus.cc:374] T b7d57b7fd5234f50afbc859bc997f3e1 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:01.153074 8050 raft_consensus.cc:729] T a4727883b2544ed58b356f41dc8017a9 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8315d22c56f7439a9e1fc82f153f018c, State: Initialized, Role: FOLLOWER
I20240501 07:59:01.153088 8051 raft_consensus.cc:729] T b7d57b7fd5234f50afbc859bc997f3e1 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b8b3ad3aa8f4053b2167dd550f2086f, State: Initialized, Role: FOLLOWER
I20240501 07:59:01.153193 8051 consensus_queue.cc:260] T b7d57b7fd5234f50afbc859bc997f3e1 P 8b8b3ad3aa8f4053b2167dd550f2086f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:59:01.153193 8050 consensus_queue.cc:260] T a4727883b2544ed58b356f41dc8017a9 P 8315d22c56f7439a9e1fc82f153f018c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:59:01.153441 8051 ts_tablet_manager.cc:1398] T b7d57b7fd5234f50afbc859bc997f3e1 P 8b8b3ad3aa8f4053b2167dd550f2086f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:01.153441 8050 ts_tablet_manager.cc:1398] T a4727883b2544ed58b356f41dc8017a9 P 8315d22c56f7439a9e1fc82f153f018c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:01.286803 7853 raft_consensus.cc:482] T d6c28cd91f624673b8e428f471a1dcce P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:01.286927 7853 raft_consensus.cc:504] T d6c28cd91f624673b8e428f471a1dcce P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } }
I20240501 07:59:01.287350 7853 leader_election.cc:288] T d6c28cd91f624673b8e428f471a1dcce P 3b164cbbc00f4e8eb3254ddd349a3ff9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129:46875), 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130:32941)
I20240501 07:59:01.287783 6446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6c28cd91f624673b8e428f471a1dcce" candidate_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8315d22c56f7439a9e1fc82f153f018c" is_pre_election: true
I20240501 07:59:01.287827 6313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6c28cd91f624673b8e428f471a1dcce" candidate_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" is_pre_election: true
I20240501 07:59:01.287914 6446 raft_consensus.cc:2454] T d6c28cd91f624673b8e428f471a1dcce P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b164cbbc00f4e8eb3254ddd349a3ff9 in term 0.
I20240501 07:59:01.287968 6313 raft_consensus.cc:2454] T d6c28cd91f624673b8e428f471a1dcce P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b164cbbc00f4e8eb3254ddd349a3ff9 in term 0.
I20240501 07:59:01.288237 6512 leader_election.cc:302] T d6c28cd91f624673b8e428f471a1dcce P 3b164cbbc00f4e8eb3254ddd349a3ff9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b164cbbc00f4e8eb3254ddd349a3ff9, 8315d22c56f7439a9e1fc82f153f018c; no voters:
I20240501 07:59:01.288429 7853 raft_consensus.cc:2781] T d6c28cd91f624673b8e428f471a1dcce P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:01.288486 7853 raft_consensus.cc:482] T d6c28cd91f624673b8e428f471a1dcce P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:01.288535 7853 raft_consensus.cc:3037] T d6c28cd91f624673b8e428f471a1dcce P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:01.289762 7853 raft_consensus.cc:504] T d6c28cd91f624673b8e428f471a1dcce P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } }
I20240501 07:59:01.290117 7853 leader_election.cc:288] T d6c28cd91f624673b8e428f471a1dcce P 3b164cbbc00f4e8eb3254ddd349a3ff9 [CANDIDATE]: Term 1 election: Requested vote from peers 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129:46875), 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130:32941)
I20240501 07:59:01.290477 6446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6c28cd91f624673b8e428f471a1dcce" candidate_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8315d22c56f7439a9e1fc82f153f018c"
I20240501 07:59:01.290582 6446 raft_consensus.cc:3037] T d6c28cd91f624673b8e428f471a1dcce P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:01.290539 6313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6c28cd91f624673b8e428f471a1dcce" candidate_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f"
I20240501 07:59:01.290663 6313 raft_consensus.cc:3037] T d6c28cd91f624673b8e428f471a1dcce P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:01.291355 6446 raft_consensus.cc:2454] T d6c28cd91f624673b8e428f471a1dcce P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b164cbbc00f4e8eb3254ddd349a3ff9 in term 1.
I20240501 07:59:01.291424 6313 raft_consensus.cc:2454] T d6c28cd91f624673b8e428f471a1dcce P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b164cbbc00f4e8eb3254ddd349a3ff9 in term 1.
I20240501 07:59:01.291638 6512 leader_election.cc:302] T d6c28cd91f624673b8e428f471a1dcce P 3b164cbbc00f4e8eb3254ddd349a3ff9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b164cbbc00f4e8eb3254ddd349a3ff9, 8315d22c56f7439a9e1fc82f153f018c; no voters:
I20240501 07:59:01.291783 7853 raft_consensus.cc:2781] T d6c28cd91f624673b8e428f471a1dcce P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:01.291954 7853 raft_consensus.cc:686] T d6c28cd91f624673b8e428f471a1dcce P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 LEADER]: Becoming Leader. State: Replica: 3b164cbbc00f4e8eb3254ddd349a3ff9, State: Running, Role: LEADER
I20240501 07:59:01.292111 7853 consensus_queue.cc:237] T d6c28cd91f624673b8e428f471a1dcce P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } }
I20240501 07:59:01.294152 6165 catalog_manager.cc:5484] T d6c28cd91f624673b8e428f471a1dcce P 3b164cbbc00f4e8eb3254ddd349a3ff9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } health_report { overall_health: HEALTHY } } }
I20240501 07:59:01.377660 7853 raft_consensus.cc:482] T b7d57b7fd5234f50afbc859bc997f3e1 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:01.377812 7853 raft_consensus.cc:504] T b7d57b7fd5234f50afbc859bc997f3e1 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:59:01.378176 7853 leader_election.cc:288] T b7d57b7fd5234f50afbc859bc997f3e1 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129:46875), 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132:36773)
I20240501 07:59:01.378577 6313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7d57b7fd5234f50afbc859bc997f3e1" candidate_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" is_pre_election: true
I20240501 07:59:01.378620 6708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7d57b7fd5234f50afbc859bc997f3e1" candidate_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f9cb72be98941b5a8e504cbe34132b9" is_pre_election: true
I20240501 07:59:01.378726 6313 raft_consensus.cc:2454] T b7d57b7fd5234f50afbc859bc997f3e1 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b164cbbc00f4e8eb3254ddd349a3ff9 in term 0.
I20240501 07:59:01.378793 6708 raft_consensus.cc:2454] T b7d57b7fd5234f50afbc859bc997f3e1 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b164cbbc00f4e8eb3254ddd349a3ff9 in term 0.
I20240501 07:59:01.379062 6510 leader_election.cc:302] T b7d57b7fd5234f50afbc859bc997f3e1 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b164cbbc00f4e8eb3254ddd349a3ff9, 8b8b3ad3aa8f4053b2167dd550f2086f; no voters:
I20240501 07:59:01.379231 7853 raft_consensus.cc:2781] T b7d57b7fd5234f50afbc859bc997f3e1 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:01.379297 7853 raft_consensus.cc:482] T b7d57b7fd5234f50afbc859bc997f3e1 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:01.379330 7853 raft_consensus.cc:3037] T b7d57b7fd5234f50afbc859bc997f3e1 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:01.380111 7853 raft_consensus.cc:504] T b7d57b7fd5234f50afbc859bc997f3e1 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:59:01.380410 7853 leader_election.cc:288] T b7d57b7fd5234f50afbc859bc997f3e1 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [CANDIDATE]: Term 1 election: Requested vote from peers 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129:46875), 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132:36773)
I20240501 07:59:01.380733 6313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7d57b7fd5234f50afbc859bc997f3e1" candidate_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f"
I20240501 07:59:01.380759 6708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7d57b7fd5234f50afbc859bc997f3e1" candidate_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f9cb72be98941b5a8e504cbe34132b9"
I20240501 07:59:01.380859 6708 raft_consensus.cc:3037] T b7d57b7fd5234f50afbc859bc997f3e1 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:01.380856 6313 raft_consensus.cc:3037] T b7d57b7fd5234f50afbc859bc997f3e1 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:01.381724 6708 raft_consensus.cc:2454] T b7d57b7fd5234f50afbc859bc997f3e1 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b164cbbc00f4e8eb3254ddd349a3ff9 in term 1.
I20240501 07:59:01.381775 6313 raft_consensus.cc:2454] T b7d57b7fd5234f50afbc859bc997f3e1 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b164cbbc00f4e8eb3254ddd349a3ff9 in term 1.
I20240501 07:59:01.382009 6512 leader_election.cc:302] T b7d57b7fd5234f50afbc859bc997f3e1 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f9cb72be98941b5a8e504cbe34132b9, 3b164cbbc00f4e8eb3254ddd349a3ff9; no voters:
I20240501 07:59:01.382167 7853 raft_consensus.cc:2781] T b7d57b7fd5234f50afbc859bc997f3e1 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:01.382256 7853 raft_consensus.cc:686] T b7d57b7fd5234f50afbc859bc997f3e1 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 LEADER]: Becoming Leader. State: Replica: 3b164cbbc00f4e8eb3254ddd349a3ff9, State: Running, Role: LEADER
I20240501 07:59:01.382413 7853 consensus_queue.cc:237] T b7d57b7fd5234f50afbc859bc997f3e1 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:59:01.384414 6165 catalog_manager.cc:5484] T b7d57b7fd5234f50afbc859bc997f3e1 P 3b164cbbc00f4e8eb3254ddd349a3ff9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } health_report { overall_health: UNKNOWN } } }
I20240501 07:59:01.484138 7853 raft_consensus.cc:482] T a4727883b2544ed58b356f41dc8017a9 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:01.484284 7853 raft_consensus.cc:504] T a4727883b2544ed58b356f41dc8017a9 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:59:01.484647 7853 leader_election.cc:288] T a4727883b2544ed58b356f41dc8017a9 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130:32941), 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132:36773)
I20240501 07:59:01.485181 6446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4727883b2544ed58b356f41dc8017a9" candidate_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8315d22c56f7439a9e1fc82f153f018c" is_pre_election: true
I20240501 07:59:01.485329 6446 raft_consensus.cc:2454] T a4727883b2544ed58b356f41dc8017a9 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b164cbbc00f4e8eb3254ddd349a3ff9 in term 0.
I20240501 07:59:01.485020 8035 raft_consensus.cc:482] T a4727883b2544ed58b356f41dc8017a9 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:01.485286 6708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4727883b2544ed58b356f41dc8017a9" candidate_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f9cb72be98941b5a8e504cbe34132b9" is_pre_election: true
I20240501 07:59:01.485605 8035 raft_consensus.cc:504] T a4727883b2544ed58b356f41dc8017a9 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:59:01.485651 6512 leader_election.cc:302] T a4727883b2544ed58b356f41dc8017a9 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b164cbbc00f4e8eb3254ddd349a3ff9, 8315d22c56f7439a9e1fc82f153f018c; no voters:
I20240501 07:59:01.485797 6708 raft_consensus.cc:2454] T a4727883b2544ed58b356f41dc8017a9 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b164cbbc00f4e8eb3254ddd349a3ff9 in term 0.
I20240501 07:59:01.485816 7853 raft_consensus.cc:2781] T a4727883b2544ed58b356f41dc8017a9 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:01.485873 7853 raft_consensus.cc:482] T a4727883b2544ed58b356f41dc8017a9 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:01.485908 7853 raft_consensus.cc:3037] T a4727883b2544ed58b356f41dc8017a9 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:01.486343 8035 leader_election.cc:288] T a4727883b2544ed58b356f41dc8017a9 P 1f9cb72be98941b5a8e504cbe34132b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131:34471), 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130:32941)
I20240501 07:59:01.486303 6577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4727883b2544ed58b356f41dc8017a9" candidate_uuid: "1f9cb72be98941b5a8e504cbe34132b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" is_pre_election: true
I20240501 07:59:01.486649 6446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4727883b2544ed58b356f41dc8017a9" candidate_uuid: "1f9cb72be98941b5a8e504cbe34132b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8315d22c56f7439a9e1fc82f153f018c" is_pre_election: true
I20240501 07:59:01.486769 6446 raft_consensus.cc:2454] T a4727883b2544ed58b356f41dc8017a9 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f9cb72be98941b5a8e504cbe34132b9 in term 0.
I20240501 07:59:01.487036 6643 leader_election.cc:302] T a4727883b2544ed58b356f41dc8017a9 P 1f9cb72be98941b5a8e504cbe34132b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f9cb72be98941b5a8e504cbe34132b9, 8315d22c56f7439a9e1fc82f153f018c; no voters:
I20240501 07:59:01.487041 7853 raft_consensus.cc:504] T a4727883b2544ed58b356f41dc8017a9 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:59:01.487234 8035 raft_consensus.cc:2781] T a4727883b2544ed58b356f41dc8017a9 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:01.487274 6577 raft_consensus.cc:2379] T a4727883b2544ed58b356f41dc8017a9 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1f9cb72be98941b5a8e504cbe34132b9 in current term 1: Already voted for candidate 3b164cbbc00f4e8eb3254ddd349a3ff9 in this term.
I20240501 07:59:01.487293 8035 raft_consensus.cc:482] T a4727883b2544ed58b356f41dc8017a9 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:01.487398 7853 leader_election.cc:288] T a4727883b2544ed58b356f41dc8017a9 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [CANDIDATE]: Term 1 election: Requested vote from peers 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130:32941), 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132:36773)
I20240501 07:59:01.487416 8035 raft_consensus.cc:3037] T a4727883b2544ed58b356f41dc8017a9 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:01.487783 6446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4727883b2544ed58b356f41dc8017a9" candidate_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8315d22c56f7439a9e1fc82f153f018c"
I20240501 07:59:01.487792 6708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4727883b2544ed58b356f41dc8017a9" candidate_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f9cb72be98941b5a8e504cbe34132b9"
I20240501 07:59:01.487890 6446 raft_consensus.cc:3037] T a4727883b2544ed58b356f41dc8017a9 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:01.488452 8035 raft_consensus.cc:504] T a4727883b2544ed58b356f41dc8017a9 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:59:01.488641 6708 raft_consensus.cc:2379] T a4727883b2544ed58b356f41dc8017a9 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 3b164cbbc00f4e8eb3254ddd349a3ff9 in current term 1: Already voted for candidate 1f9cb72be98941b5a8e504cbe34132b9 in this term.
I20240501 07:59:01.488737 6446 raft_consensus.cc:2454] T a4727883b2544ed58b356f41dc8017a9 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b164cbbc00f4e8eb3254ddd349a3ff9 in term 1.
I20240501 07:59:01.489061 6512 leader_election.cc:302] T a4727883b2544ed58b356f41dc8017a9 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b164cbbc00f4e8eb3254ddd349a3ff9, 8315d22c56f7439a9e1fc82f153f018c; no voters: 1f9cb72be98941b5a8e504cbe34132b9
I20240501 07:59:01.489102 8035 leader_election.cc:288] T a4727883b2544ed58b356f41dc8017a9 P 1f9cb72be98941b5a8e504cbe34132b9 [CANDIDATE]: Term 1 election: Requested vote from peers 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131:34471), 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130:32941)
I20240501 07:59:01.489440 6577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4727883b2544ed58b356f41dc8017a9" candidate_uuid: "1f9cb72be98941b5a8e504cbe34132b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9"
I20240501 07:59:01.489573 6577 raft_consensus.cc:2379] T a4727883b2544ed58b356f41dc8017a9 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 1f9cb72be98941b5a8e504cbe34132b9 in current term 1: Already voted for candidate 3b164cbbc00f4e8eb3254ddd349a3ff9 in this term.
I20240501 07:59:01.489869 7853 raft_consensus.cc:2781] T a4727883b2544ed58b356f41dc8017a9 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:01.489964 7853 raft_consensus.cc:686] T a4727883b2544ed58b356f41dc8017a9 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 LEADER]: Becoming Leader. State: Replica: 3b164cbbc00f4e8eb3254ddd349a3ff9, State: Running, Role: LEADER
I20240501 07:59:01.490015 6446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4727883b2544ed58b356f41dc8017a9" candidate_uuid: "1f9cb72be98941b5a8e504cbe34132b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8315d22c56f7439a9e1fc82f153f018c"
I20240501 07:59:01.490136 6446 raft_consensus.cc:2379] T a4727883b2544ed58b356f41dc8017a9 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 1f9cb72be98941b5a8e504cbe34132b9 in current term 1: Already voted for candidate 3b164cbbc00f4e8eb3254ddd349a3ff9 in this term.
I20240501 07:59:01.490093 7853 consensus_queue.cc:237] T a4727883b2544ed58b356f41dc8017a9 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:59:01.490381 6643 leader_election.cc:302] T a4727883b2544ed58b356f41dc8017a9 P 1f9cb72be98941b5a8e504cbe34132b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1f9cb72be98941b5a8e504cbe34132b9; no voters: 3b164cbbc00f4e8eb3254ddd349a3ff9, 8315d22c56f7439a9e1fc82f153f018c
I20240501 07:59:01.490526 7959 raft_consensus.cc:2726] T a4727883b2544ed58b356f41dc8017a9 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240501 07:59:01.492026 6165 catalog_manager.cc:5484] T a4727883b2544ed58b356f41dc8017a9 P 3b164cbbc00f4e8eb3254ddd349a3ff9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } health_report { overall_health: UNKNOWN } } }
W20240501 07:59:01.619625 8119 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:59:01.619918 8119 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:59:01.623531 8119 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_11'
I20240501 07:59:01.704900 6708 raft_consensus.cc:1261] T b7d57b7fd5234f50afbc859bc997f3e1 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Refusing update from remote peer 3b164cbbc00f4e8eb3254ddd349a3ff9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:01.704900 6314 raft_consensus.cc:1261] T b7d57b7fd5234f50afbc859bc997f3e1 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Refusing update from remote peer 3b164cbbc00f4e8eb3254ddd349a3ff9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:01.705442 7853 consensus_queue.cc:1035] T b7d57b7fd5234f50afbc859bc997f3e1 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:01.705722 6446 raft_consensus.cc:1261] T d6c28cd91f624673b8e428f471a1dcce P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Refusing update from remote peer 3b164cbbc00f4e8eb3254ddd349a3ff9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:01.705756 7853 consensus_queue.cc:1035] T b7d57b7fd5234f50afbc859bc997f3e1 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:01.706215 7853 consensus_queue.cc:1035] T d6c28cd91f624673b8e428f471a1dcce P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:01.709391 6313 raft_consensus.cc:1261] T d6c28cd91f624673b8e428f471a1dcce P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Refusing update from remote peer 3b164cbbc00f4e8eb3254ddd349a3ff9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:01.709435 6445 raft_consensus.cc:1261] T a4727883b2544ed58b356f41dc8017a9 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Refusing update from remote peer 3b164cbbc00f4e8eb3254ddd349a3ff9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:01.709818 7853 consensus_queue.cc:1035] T a4727883b2544ed58b356f41dc8017a9 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:01.710047 8115 consensus_queue.cc:1035] T d6c28cd91f624673b8e428f471a1dcce P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:01.712841 6707 raft_consensus.cc:1261] T a4727883b2544ed58b356f41dc8017a9 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Refusing update from remote peer 3b164cbbc00f4e8eb3254ddd349a3ff9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:01.715571 8115 consensus_queue.cc:1035] T a4727883b2544ed58b356f41dc8017a9 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 97.6099 ms
time per row: 0.048805 ms
I20240501 07:59:01.769230 6170 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55950:
name: "rebalance_test_table_12"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 07:59:01.775854 6557 tablet_service.cc:1467] Processing CreateTablet for tablet bab34f083c8045e4a6df373db5ebd54b (DEFAULT_TABLE table=rebalance_test_table_12 [id=f041050203204d0690b4b8f368f834b2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:01.775844 6426 tablet_service.cc:1467] Processing CreateTablet for tablet bab34f083c8045e4a6df373db5ebd54b (DEFAULT_TABLE table=rebalance_test_table_12 [id=f041050203204d0690b4b8f368f834b2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:01.776193 6426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bab34f083c8045e4a6df373db5ebd54b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:01.776196 6557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bab34f083c8045e4a6df373db5ebd54b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:01.776319 6293 tablet_service.cc:1467] Processing CreateTablet for tablet bab34f083c8045e4a6df373db5ebd54b (DEFAULT_TABLE table=rebalance_test_table_12 [id=f041050203204d0690b4b8f368f834b2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:01.776587 6293 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bab34f083c8045e4a6df373db5ebd54b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:01.778878 8180 tablet_bootstrap.cc:492] T bab34f083c8045e4a6df373db5ebd54b P 3b164cbbc00f4e8eb3254ddd349a3ff9: Bootstrap starting.
I20240501 07:59:01.779338 6294 tablet_service.cc:1467] Processing CreateTablet for tablet 0b680bccf110443e9495942863364ff9 (DEFAULT_TABLE table=rebalance_test_table_12 [id=f041050203204d0690b4b8f368f834b2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:01.779348 6686 tablet_service.cc:1467] Processing CreateTablet for tablet adf207c019994dd49671bab4145fc268 (DEFAULT_TABLE table=rebalance_test_table_12 [id=f041050203204d0690b4b8f368f834b2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:01.779456 8179 tablet_bootstrap.cc:492] T bab34f083c8045e4a6df373db5ebd54b P 8315d22c56f7439a9e1fc82f153f018c: Bootstrap starting.
I20240501 07:59:01.779577 6294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b680bccf110443e9495942863364ff9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:01.779577 6686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet adf207c019994dd49671bab4145fc268. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:01.779951 8180 tablet_bootstrap.cc:654] T bab34f083c8045e4a6df373db5ebd54b P 3b164cbbc00f4e8eb3254ddd349a3ff9: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:01.780391 8179 tablet_bootstrap.cc:654] T bab34f083c8045e4a6df373db5ebd54b P 8315d22c56f7439a9e1fc82f153f018c: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:01.780993 8181 tablet_bootstrap.cc:492] T bab34f083c8045e4a6df373db5ebd54b P 8b8b3ad3aa8f4053b2167dd550f2086f: Bootstrap starting.
I20240501 07:59:01.781137 6687 tablet_service.cc:1467] Processing CreateTablet for tablet 0b680bccf110443e9495942863364ff9 (DEFAULT_TABLE table=rebalance_test_table_12 [id=f041050203204d0690b4b8f368f834b2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:01.781394 6687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b680bccf110443e9495942863364ff9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:01.782166 6293 tablet_service.cc:1467] Processing CreateTablet for tablet adf207c019994dd49671bab4145fc268 (DEFAULT_TABLE table=rebalance_test_table_12 [id=f041050203204d0690b4b8f368f834b2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:01.782243 8181 tablet_bootstrap.cc:654] T bab34f083c8045e4a6df373db5ebd54b P 8b8b3ad3aa8f4053b2167dd550f2086f: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:01.782388 6293 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet adf207c019994dd49671bab4145fc268. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:01.782392 6426 tablet_service.cc:1467] Processing CreateTablet for tablet adf207c019994dd49671bab4145fc268 (DEFAULT_TABLE table=rebalance_test_table_12 [id=f041050203204d0690b4b8f368f834b2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:01.782620 6426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet adf207c019994dd49671bab4145fc268. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:01.783448 6557 tablet_service.cc:1467] Processing CreateTablet for tablet 0b680bccf110443e9495942863364ff9 (DEFAULT_TABLE table=rebalance_test_table_12 [id=f041050203204d0690b4b8f368f834b2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:01.783736 6557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b680bccf110443e9495942863364ff9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:01.784219 8183 tablet_bootstrap.cc:492] T adf207c019994dd49671bab4145fc268 P 1f9cb72be98941b5a8e504cbe34132b9: Bootstrap starting.
I20240501 07:59:01.784655 8180 tablet_bootstrap.cc:492] T bab34f083c8045e4a6df373db5ebd54b P 3b164cbbc00f4e8eb3254ddd349a3ff9: No bootstrap required, opened a new log
I20240501 07:59:01.784755 8180 ts_tablet_manager.cc:1367] T bab34f083c8045e4a6df373db5ebd54b P 3b164cbbc00f4e8eb3254ddd349a3ff9: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 07:59:01.785106 8183 tablet_bootstrap.cc:654] T adf207c019994dd49671bab4145fc268 P 1f9cb72be98941b5a8e504cbe34132b9: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:01.785481 8180 raft_consensus.cc:348] T bab34f083c8045e4a6df373db5ebd54b P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } }
I20240501 07:59:01.785604 8180 raft_consensus.cc:374] T bab34f083c8045e4a6df373db5ebd54b P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:01.785643 8180 raft_consensus.cc:729] T bab34f083c8045e4a6df373db5ebd54b P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b164cbbc00f4e8eb3254ddd349a3ff9, State: Initialized, Role: FOLLOWER
I20240501 07:59:01.785812 8180 consensus_queue.cc:260] T bab34f083c8045e4a6df373db5ebd54b P 3b164cbbc00f4e8eb3254ddd349a3ff9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } }
I20240501 07:59:01.786320 8180 ts_tablet_manager.cc:1398] T bab34f083c8045e4a6df373db5ebd54b P 3b164cbbc00f4e8eb3254ddd349a3ff9: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:59:01.786595 8181 tablet_bootstrap.cc:492] T bab34f083c8045e4a6df373db5ebd54b P 8b8b3ad3aa8f4053b2167dd550f2086f: No bootstrap required, opened a new log
I20240501 07:59:01.786711 8181 ts_tablet_manager.cc:1367] T bab34f083c8045e4a6df373db5ebd54b P 8b8b3ad3aa8f4053b2167dd550f2086f: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 07:59:01.787254 8181 raft_consensus.cc:348] T bab34f083c8045e4a6df373db5ebd54b P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } }
I20240501 07:59:01.787372 8181 raft_consensus.cc:374] T bab34f083c8045e4a6df373db5ebd54b P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:01.787410 8181 raft_consensus.cc:729] T bab34f083c8045e4a6df373db5ebd54b P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b8b3ad3aa8f4053b2167dd550f2086f, State: Initialized, Role: FOLLOWER
I20240501 07:59:01.787532 8181 consensus_queue.cc:260] T bab34f083c8045e4a6df373db5ebd54b P 8b8b3ad3aa8f4053b2167dd550f2086f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } }
I20240501 07:59:01.787817 8181 ts_tablet_manager.cc:1398] T bab34f083c8045e4a6df373db5ebd54b P 8b8b3ad3aa8f4053b2167dd550f2086f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:01.787698 8180 tablet_bootstrap.cc:492] T 0b680bccf110443e9495942863364ff9 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Bootstrap starting.
I20240501 07:59:01.787988 8181 tablet_bootstrap.cc:492] T 0b680bccf110443e9495942863364ff9 P 8b8b3ad3aa8f4053b2167dd550f2086f: Bootstrap starting.
I20240501 07:59:01.788918 8181 tablet_bootstrap.cc:654] T 0b680bccf110443e9495942863364ff9 P 8b8b3ad3aa8f4053b2167dd550f2086f: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:01.789038 8180 tablet_bootstrap.cc:654] T 0b680bccf110443e9495942863364ff9 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:01.789634 8179 tablet_bootstrap.cc:492] T bab34f083c8045e4a6df373db5ebd54b P 8315d22c56f7439a9e1fc82f153f018c: No bootstrap required, opened a new log
I20240501 07:59:01.789749 8179 ts_tablet_manager.cc:1367] T bab34f083c8045e4a6df373db5ebd54b P 8315d22c56f7439a9e1fc82f153f018c: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.000s
I20240501 07:59:01.790102 8180 tablet_bootstrap.cc:492] T 0b680bccf110443e9495942863364ff9 P 3b164cbbc00f4e8eb3254ddd349a3ff9: No bootstrap required, opened a new log
I20240501 07:59:01.790172 8180 ts_tablet_manager.cc:1367] T 0b680bccf110443e9495942863364ff9 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 07:59:01.790234 8179 raft_consensus.cc:348] T bab34f083c8045e4a6df373db5ebd54b P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } }
I20240501 07:59:01.790342 8179 raft_consensus.cc:374] T bab34f083c8045e4a6df373db5ebd54b P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:01.790372 8179 raft_consensus.cc:729] T bab34f083c8045e4a6df373db5ebd54b P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8315d22c56f7439a9e1fc82f153f018c, State: Initialized, Role: FOLLOWER
I20240501 07:59:01.790493 8179 consensus_queue.cc:260] T bab34f083c8045e4a6df373db5ebd54b P 8315d22c56f7439a9e1fc82f153f018c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } }
I20240501 07:59:01.790618 8180 raft_consensus.cc:348] T 0b680bccf110443e9495942863364ff9 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } }
I20240501 07:59:01.790704 8180 raft_consensus.cc:374] T 0b680bccf110443e9495942863364ff9 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:01.790732 8180 raft_consensus.cc:729] T 0b680bccf110443e9495942863364ff9 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b164cbbc00f4e8eb3254ddd349a3ff9, State: Initialized, Role: FOLLOWER
I20240501 07:59:01.790779 8179 ts_tablet_manager.cc:1398] T bab34f083c8045e4a6df373db5ebd54b P 8315d22c56f7439a9e1fc82f153f018c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:01.790840 8180 consensus_queue.cc:260] T 0b680bccf110443e9495942863364ff9 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } }
I20240501 07:59:01.790930 8179 tablet_bootstrap.cc:492] T adf207c019994dd49671bab4145fc268 P 8315d22c56f7439a9e1fc82f153f018c: Bootstrap starting.
I20240501 07:59:01.791095 8180 ts_tablet_manager.cc:1398] T 0b680bccf110443e9495942863364ff9 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:01.791604 8181 tablet_bootstrap.cc:492] T 0b680bccf110443e9495942863364ff9 P 8b8b3ad3aa8f4053b2167dd550f2086f: No bootstrap required, opened a new log
I20240501 07:59:01.791687 8181 ts_tablet_manager.cc:1367] T 0b680bccf110443e9495942863364ff9 P 8b8b3ad3aa8f4053b2167dd550f2086f: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20240501 07:59:01.791836 8179 tablet_bootstrap.cc:654] T adf207c019994dd49671bab4145fc268 P 8315d22c56f7439a9e1fc82f153f018c: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:01.792147 8181 raft_consensus.cc:348] T 0b680bccf110443e9495942863364ff9 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } }
I20240501 07:59:01.792253 8181 raft_consensus.cc:374] T 0b680bccf110443e9495942863364ff9 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:01.792282 8181 raft_consensus.cc:729] T 0b680bccf110443e9495942863364ff9 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b8b3ad3aa8f4053b2167dd550f2086f, State: Initialized, Role: FOLLOWER
I20240501 07:59:01.792389 8181 consensus_queue.cc:260] T 0b680bccf110443e9495942863364ff9 P 8b8b3ad3aa8f4053b2167dd550f2086f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } }
I20240501 07:59:01.792685 8181 ts_tablet_manager.cc:1398] T 0b680bccf110443e9495942863364ff9 P 8b8b3ad3aa8f4053b2167dd550f2086f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:59:01.792860 8181 tablet_bootstrap.cc:492] T adf207c019994dd49671bab4145fc268 P 8b8b3ad3aa8f4053b2167dd550f2086f: Bootstrap starting.
I20240501 07:59:01.793133 8183 tablet_bootstrap.cc:492] T adf207c019994dd49671bab4145fc268 P 1f9cb72be98941b5a8e504cbe34132b9: No bootstrap required, opened a new log
I20240501 07:59:01.793223 8183 ts_tablet_manager.cc:1367] T adf207c019994dd49671bab4145fc268 P 1f9cb72be98941b5a8e504cbe34132b9: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.002s
I20240501 07:59:01.793797 8181 tablet_bootstrap.cc:654] T adf207c019994dd49671bab4145fc268 P 8b8b3ad3aa8f4053b2167dd550f2086f: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:01.794004 8183 raft_consensus.cc:348] T adf207c019994dd49671bab4145fc268 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:59:01.794106 8183 raft_consensus.cc:374] T adf207c019994dd49671bab4145fc268 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:01.794142 8183 raft_consensus.cc:729] T adf207c019994dd49671bab4145fc268 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f9cb72be98941b5a8e504cbe34132b9, State: Initialized, Role: FOLLOWER
I20240501 07:59:01.794263 8183 consensus_queue.cc:260] T adf207c019994dd49671bab4145fc268 P 1f9cb72be98941b5a8e504cbe34132b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:59:01.794636 8183 ts_tablet_manager.cc:1398] T adf207c019994dd49671bab4145fc268 P 1f9cb72be98941b5a8e504cbe34132b9: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 07:59:01.794873 8183 tablet_bootstrap.cc:492] T 0b680bccf110443e9495942863364ff9 P 1f9cb72be98941b5a8e504cbe34132b9: Bootstrap starting.
I20240501 07:59:01.795764 8181 tablet_bootstrap.cc:492] T adf207c019994dd49671bab4145fc268 P 8b8b3ad3aa8f4053b2167dd550f2086f: No bootstrap required, opened a new log
I20240501 07:59:01.795846 8181 ts_tablet_manager.cc:1367] T adf207c019994dd49671bab4145fc268 P 8b8b3ad3aa8f4053b2167dd550f2086f: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240501 07:59:01.796247 8181 raft_consensus.cc:348] T adf207c019994dd49671bab4145fc268 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:59:01.796360 8181 raft_consensus.cc:374] T adf207c019994dd49671bab4145fc268 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:01.796397 8181 raft_consensus.cc:729] T adf207c019994dd49671bab4145fc268 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b8b3ad3aa8f4053b2167dd550f2086f, State: Initialized, Role: FOLLOWER
I20240501 07:59:01.796479 8179 tablet_bootstrap.cc:492] T adf207c019994dd49671bab4145fc268 P 8315d22c56f7439a9e1fc82f153f018c: No bootstrap required, opened a new log
I20240501 07:59:01.796555 8179 ts_tablet_manager.cc:1367] T adf207c019994dd49671bab4145fc268 P 8315d22c56f7439a9e1fc82f153f018c: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 07:59:01.796522 8181 consensus_queue.cc:260] T adf207c019994dd49671bab4145fc268 P 8b8b3ad3aa8f4053b2167dd550f2086f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:59:01.796787 8181 ts_tablet_manager.cc:1398] T adf207c019994dd49671bab4145fc268 P 8b8b3ad3aa8f4053b2167dd550f2086f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:01.797168 8179 raft_consensus.cc:348] T adf207c019994dd49671bab4145fc268 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:59:01.797326 8179 raft_consensus.cc:374] T adf207c019994dd49671bab4145fc268 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:01.797410 8179 raft_consensus.cc:729] T adf207c019994dd49671bab4145fc268 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8315d22c56f7439a9e1fc82f153f018c, State: Initialized, Role: FOLLOWER
I20240501 07:59:01.797602 8179 consensus_queue.cc:260] T adf207c019994dd49671bab4145fc268 P 8315d22c56f7439a9e1fc82f153f018c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:59:01.797926 8179 ts_tablet_manager.cc:1398] T adf207c019994dd49671bab4145fc268 P 8315d22c56f7439a9e1fc82f153f018c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:01.799748 8183 tablet_bootstrap.cc:654] T 0b680bccf110443e9495942863364ff9 P 1f9cb72be98941b5a8e504cbe34132b9: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:01.800827 8183 tablet_bootstrap.cc:492] T 0b680bccf110443e9495942863364ff9 P 1f9cb72be98941b5a8e504cbe34132b9: No bootstrap required, opened a new log
I20240501 07:59:01.800895 8183 ts_tablet_manager.cc:1367] T 0b680bccf110443e9495942863364ff9 P 1f9cb72be98941b5a8e504cbe34132b9: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.002s
I20240501 07:59:01.801301 8183 raft_consensus.cc:348] T 0b680bccf110443e9495942863364ff9 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } }
I20240501 07:59:01.801421 8183 raft_consensus.cc:374] T 0b680bccf110443e9495942863364ff9 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:01.801450 8183 raft_consensus.cc:729] T 0b680bccf110443e9495942863364ff9 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f9cb72be98941b5a8e504cbe34132b9, State: Initialized, Role: FOLLOWER
I20240501 07:59:01.801560 8183 consensus_queue.cc:260] T 0b680bccf110443e9495942863364ff9 P 1f9cb72be98941b5a8e504cbe34132b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } }
I20240501 07:59:01.801801 8183 ts_tablet_manager.cc:1398] T 0b680bccf110443e9495942863364ff9 P 1f9cb72be98941b5a8e504cbe34132b9: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:59:01.804819 8166 raft_consensus.cc:482] T 0b680bccf110443e9495942863364ff9 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:01.804916 8166 raft_consensus.cc:504] T 0b680bccf110443e9495942863364ff9 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } }
I20240501 07:59:01.805380 8166 leader_election.cc:288] T 0b680bccf110443e9495942863364ff9 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132:36773), 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129:46875)
I20240501 07:59:01.805649 6707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b680bccf110443e9495942863364ff9" candidate_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f9cb72be98941b5a8e504cbe34132b9" is_pre_election: true
I20240501 07:59:01.805750 6707 raft_consensus.cc:2454] T 0b680bccf110443e9495942863364ff9 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b164cbbc00f4e8eb3254ddd349a3ff9 in term 0.
I20240501 07:59:01.805737 6314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b680bccf110443e9495942863364ff9" candidate_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" is_pre_election: true
I20240501 07:59:01.805863 6314 raft_consensus.cc:2454] T 0b680bccf110443e9495942863364ff9 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b164cbbc00f4e8eb3254ddd349a3ff9 in term 0.
I20240501 07:59:01.806035 6512 leader_election.cc:302] T 0b680bccf110443e9495942863364ff9 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f9cb72be98941b5a8e504cbe34132b9, 3b164cbbc00f4e8eb3254ddd349a3ff9; no voters:
I20240501 07:59:01.806187 8166 raft_consensus.cc:2781] T 0b680bccf110443e9495942863364ff9 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:01.806242 8166 raft_consensus.cc:482] T 0b680bccf110443e9495942863364ff9 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:01.806275 8166 raft_consensus.cc:3037] T 0b680bccf110443e9495942863364ff9 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:01.807004 8166 raft_consensus.cc:504] T 0b680bccf110443e9495942863364ff9 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } }
I20240501 07:59:01.807286 8166 leader_election.cc:288] T 0b680bccf110443e9495942863364ff9 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [CANDIDATE]: Term 1 election: Requested vote from peers 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132:36773), 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129:46875)
I20240501 07:59:01.807590 6314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b680bccf110443e9495942863364ff9" candidate_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f"
I20240501 07:59:01.807590 6707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b680bccf110443e9495942863364ff9" candidate_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f9cb72be98941b5a8e504cbe34132b9"
I20240501 07:59:01.807709 6314 raft_consensus.cc:3037] T 0b680bccf110443e9495942863364ff9 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:01.807741 6707 raft_consensus.cc:3037] T 0b680bccf110443e9495942863364ff9 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:01.808670 6314 raft_consensus.cc:2454] T 0b680bccf110443e9495942863364ff9 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b164cbbc00f4e8eb3254ddd349a3ff9 in term 1.
I20240501 07:59:01.808670 6707 raft_consensus.cc:2454] T 0b680bccf110443e9495942863364ff9 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b164cbbc00f4e8eb3254ddd349a3ff9 in term 1.
I20240501 07:59:01.808951 6512 leader_election.cc:302] T 0b680bccf110443e9495942863364ff9 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f9cb72be98941b5a8e504cbe34132b9, 3b164cbbc00f4e8eb3254ddd349a3ff9; no voters:
I20240501 07:59:01.809132 8166 raft_consensus.cc:2781] T 0b680bccf110443e9495942863364ff9 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:01.809216 8166 raft_consensus.cc:686] T 0b680bccf110443e9495942863364ff9 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 LEADER]: Becoming Leader. State: Replica: 3b164cbbc00f4e8eb3254ddd349a3ff9, State: Running, Role: LEADER
I20240501 07:59:01.809351 8166 consensus_queue.cc:237] T 0b680bccf110443e9495942863364ff9 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } }
I20240501 07:59:01.811268 6170 catalog_manager.cc:5484] T 0b680bccf110443e9495942863364ff9 P 3b164cbbc00f4e8eb3254ddd349a3ff9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } health_report { overall_health: UNKNOWN } } }
I20240501 07:59:01.862753 8045 raft_consensus.cc:482] T adf207c019994dd49671bab4145fc268 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:01.862893 8045 raft_consensus.cc:504] T adf207c019994dd49671bab4145fc268 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:59:01.863317 8045 leader_election.cc:288] T adf207c019994dd49671bab4145fc268 P 8315d22c56f7439a9e1fc82f153f018c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129:46875), 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132:36773)
I20240501 07:59:01.863742 6707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adf207c019994dd49671bab4145fc268" candidate_uuid: "8315d22c56f7439a9e1fc82f153f018c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f9cb72be98941b5a8e504cbe34132b9" is_pre_election: true
I20240501 07:59:01.863726 6314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adf207c019994dd49671bab4145fc268" candidate_uuid: "8315d22c56f7439a9e1fc82f153f018c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" is_pre_election: true
I20240501 07:59:01.863893 6314 raft_consensus.cc:2454] T adf207c019994dd49671bab4145fc268 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8315d22c56f7439a9e1fc82f153f018c in term 0.
I20240501 07:59:01.863895 6707 raft_consensus.cc:2454] T adf207c019994dd49671bab4145fc268 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8315d22c56f7439a9e1fc82f153f018c in term 0.
I20240501 07:59:01.864202 6381 leader_election.cc:302] T adf207c019994dd49671bab4145fc268 P 8315d22c56f7439a9e1fc82f153f018c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f9cb72be98941b5a8e504cbe34132b9, 8315d22c56f7439a9e1fc82f153f018c; no voters:
I20240501 07:59:01.864357 8045 raft_consensus.cc:2781] T adf207c019994dd49671bab4145fc268 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:01.864416 8045 raft_consensus.cc:482] T adf207c019994dd49671bab4145fc268 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:01.864471 8045 raft_consensus.cc:3037] T adf207c019994dd49671bab4145fc268 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:01.865482 8045 raft_consensus.cc:504] T adf207c019994dd49671bab4145fc268 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:59:01.865803 8045 leader_election.cc:288] T adf207c019994dd49671bab4145fc268 P 8315d22c56f7439a9e1fc82f153f018c [CANDIDATE]: Term 1 election: Requested vote from peers 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129:46875), 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132:36773)
I20240501 07:59:01.866075 6314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adf207c019994dd49671bab4145fc268" candidate_uuid: "8315d22c56f7439a9e1fc82f153f018c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f"
I20240501 07:59:01.866103 6707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adf207c019994dd49671bab4145fc268" candidate_uuid: "8315d22c56f7439a9e1fc82f153f018c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f9cb72be98941b5a8e504cbe34132b9"
I20240501 07:59:01.866180 6314 raft_consensus.cc:3037] T adf207c019994dd49671bab4145fc268 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:01.866225 6707 raft_consensus.cc:3037] T adf207c019994dd49671bab4145fc268 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:01.866948 6314 raft_consensus.cc:2454] T adf207c019994dd49671bab4145fc268 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8315d22c56f7439a9e1fc82f153f018c in term 1.
I20240501 07:59:01.867043 6707 raft_consensus.cc:2454] T adf207c019994dd49671bab4145fc268 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8315d22c56f7439a9e1fc82f153f018c in term 1.
I20240501 07:59:01.867228 6379 leader_election.cc:302] T adf207c019994dd49671bab4145fc268 P 8315d22c56f7439a9e1fc82f153f018c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8315d22c56f7439a9e1fc82f153f018c, 8b8b3ad3aa8f4053b2167dd550f2086f; no voters:
I20240501 07:59:01.867406 8045 raft_consensus.cc:2781] T adf207c019994dd49671bab4145fc268 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:01.867617 8045 raft_consensus.cc:686] T adf207c019994dd49671bab4145fc268 P 8315d22c56f7439a9e1fc82f153f018c [term 1 LEADER]: Becoming Leader. State: Replica: 8315d22c56f7439a9e1fc82f153f018c, State: Running, Role: LEADER
I20240501 07:59:01.867763 8045 consensus_queue.cc:237] T adf207c019994dd49671bab4145fc268 P 8315d22c56f7439a9e1fc82f153f018c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:59:01.869531 6170 catalog_manager.cc:5484] T adf207c019994dd49671bab4145fc268 P 8315d22c56f7439a9e1fc82f153f018c reported cstate change: term changed from 0 to 1, leader changed from <none> to 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130). New cstate: current_term: 1 leader_uuid: "8315d22c56f7439a9e1fc82f153f018c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } health_report { overall_health: UNKNOWN } } }
I20240501 07:59:02.029457 8045 raft_consensus.cc:482] T bab34f083c8045e4a6df373db5ebd54b P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:02.029582 8045 raft_consensus.cc:504] T bab34f083c8045e4a6df373db5ebd54b P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } }
I20240501 07:59:02.030329 6314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bab34f083c8045e4a6df373db5ebd54b" candidate_uuid: "8315d22c56f7439a9e1fc82f153f018c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" is_pre_election: true
I20240501 07:59:02.030361 6577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bab34f083c8045e4a6df373db5ebd54b" candidate_uuid: "8315d22c56f7439a9e1fc82f153f018c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" is_pre_election: true
I20240501 07:59:02.030473 6577 raft_consensus.cc:2454] T bab34f083c8045e4a6df373db5ebd54b P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8315d22c56f7439a9e1fc82f153f018c in term 0.
I20240501 07:59:02.030484 6314 raft_consensus.cc:2454] T bab34f083c8045e4a6df373db5ebd54b P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8315d22c56f7439a9e1fc82f153f018c in term 0.
I20240501 07:59:02.030261 8045 leader_election.cc:288] T bab34f083c8045e4a6df373db5ebd54b P 8315d22c56f7439a9e1fc82f153f018c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131:34471), 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129:46875)
I20240501 07:59:02.030850 6382 leader_election.cc:302] T bab34f083c8045e4a6df373db5ebd54b P 8315d22c56f7439a9e1fc82f153f018c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b164cbbc00f4e8eb3254ddd349a3ff9, 8315d22c56f7439a9e1fc82f153f018c; no voters:
I20240501 07:59:02.031075 8045 raft_consensus.cc:2781] T bab34f083c8045e4a6df373db5ebd54b P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:02.031137 8045 raft_consensus.cc:482] T bab34f083c8045e4a6df373db5ebd54b P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:02.031181 8045 raft_consensus.cc:3037] T bab34f083c8045e4a6df373db5ebd54b P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:02.032161 8045 raft_consensus.cc:504] T bab34f083c8045e4a6df373db5ebd54b P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } }
I20240501 07:59:02.032573 8045 leader_election.cc:288] T bab34f083c8045e4a6df373db5ebd54b P 8315d22c56f7439a9e1fc82f153f018c [CANDIDATE]: Term 1 election: Requested vote from peers 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131:34471), 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129:46875)
I20240501 07:59:02.032805 6577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bab34f083c8045e4a6df373db5ebd54b" candidate_uuid: "8315d22c56f7439a9e1fc82f153f018c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9"
I20240501 07:59:02.032843 6314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bab34f083c8045e4a6df373db5ebd54b" candidate_uuid: "8315d22c56f7439a9e1fc82f153f018c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f"
I20240501 07:59:02.032927 6577 raft_consensus.cc:3037] T bab34f083c8045e4a6df373db5ebd54b P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:02.032932 6314 raft_consensus.cc:3037] T bab34f083c8045e4a6df373db5ebd54b P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:02.033883 6314 raft_consensus.cc:2454] T bab34f083c8045e4a6df373db5ebd54b P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8315d22c56f7439a9e1fc82f153f018c in term 1.
I20240501 07:59:02.033883 6577 raft_consensus.cc:2454] T bab34f083c8045e4a6df373db5ebd54b P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8315d22c56f7439a9e1fc82f153f018c in term 1.
I20240501 07:59:02.034158 6382 leader_election.cc:302] T bab34f083c8045e4a6df373db5ebd54b P 8315d22c56f7439a9e1fc82f153f018c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b164cbbc00f4e8eb3254ddd349a3ff9, 8315d22c56f7439a9e1fc82f153f018c; no voters:
I20240501 07:59:02.034346 8045 raft_consensus.cc:2781] T bab34f083c8045e4a6df373db5ebd54b P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:02.034427 8045 raft_consensus.cc:686] T bab34f083c8045e4a6df373db5ebd54b P 8315d22c56f7439a9e1fc82f153f018c [term 1 LEADER]: Becoming Leader. State: Replica: 8315d22c56f7439a9e1fc82f153f018c, State: Running, Role: LEADER
I20240501 07:59:02.034544 8045 consensus_queue.cc:237] T bab34f083c8045e4a6df373db5ebd54b P 8315d22c56f7439a9e1fc82f153f018c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } }
I20240501 07:59:02.036336 6170 catalog_manager.cc:5484] T bab34f083c8045e4a6df373db5ebd54b P 8315d22c56f7439a9e1fc82f153f018c reported cstate change: term changed from 0 to 1, leader changed from <none> to 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130). New cstate: current_term: 1 leader_uuid: "8315d22c56f7439a9e1fc82f153f018c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } health_report { overall_health: UNKNOWN } } }
W20240501 07:59:02.191717 8196 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:59:02.191944 8196 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:59:02.195013 8196 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_12'
I20240501 07:59:02.252290 8166 consensus_queue.cc:1035] T 0b680bccf110443e9495942863364ff9 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:02.256460 8166 consensus_queue.cc:1035] T 0b680bccf110443e9495942863364ff9 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:02.289332 6707 raft_consensus.cc:1261] T adf207c019994dd49671bab4145fc268 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Refusing update from remote peer 8315d22c56f7439a9e1fc82f153f018c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:02.289462 6314 raft_consensus.cc:1261] T bab34f083c8045e4a6df373db5ebd54b P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Refusing update from remote peer 8315d22c56f7439a9e1fc82f153f018c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:02.289513 6313 raft_consensus.cc:1261] T adf207c019994dd49671bab4145fc268 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Refusing update from remote peer 8315d22c56f7439a9e1fc82f153f018c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:02.290323 8234 consensus_queue.cc:1035] T bab34f083c8045e4a6df373db5ebd54b P 8315d22c56f7439a9e1fc82f153f018c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:02.290405 8193 consensus_queue.cc:1035] T adf207c019994dd49671bab4145fc268 P 8315d22c56f7439a9e1fc82f153f018c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:02.291065 6577 raft_consensus.cc:1261] T bab34f083c8045e4a6df373db5ebd54b P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Refusing update from remote peer 8315d22c56f7439a9e1fc82f153f018c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:02.291456 8045 consensus_queue.cc:1035] T bab34f083c8045e4a6df373db5ebd54b P 8315d22c56f7439a9e1fc82f153f018c [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:02.296000 8045 consensus_queue.cc:1035] T adf207c019994dd49671bab4145fc268 P 8315d22c56f7439a9e1fc82f153f018c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:02.328783 8222 mvcc.cc:205] Tried to move back new op lower bound from 7022798202012422144 to 7022798200974782464. Current Snapshot: MvccSnapshot[applied={T|T < 7022798202012422144}]
I20240501 07:59:02.349074 8233 mvcc.cc:205] Tried to move back new op lower bound from 7022798202190114816 to 7022798200974782464. Current Snapshot: MvccSnapshot[applied={T|T < 7022798202190114816}]
INSERT report
rows total: 2000
time total: 121.007 ms
time per row: 0.0605037 ms
I20240501 07:59:02.362284 6169 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55950:
name: "rebalance_test_table_13"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 07:59:02.369045 6557 tablet_service.cc:1467] Processing CreateTablet for tablet d3f2250a62c6429e886753aaffcbe883 (DEFAULT_TABLE table=rebalance_test_table_13 [id=886e8147658c4cffb480ca3fb821c563]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:02.369129 6426 tablet_service.cc:1467] Processing CreateTablet for tablet d3f2250a62c6429e886753aaffcbe883 (DEFAULT_TABLE table=rebalance_test_table_13 [id=886e8147658c4cffb480ca3fb821c563]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:02.369369 6557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3f2250a62c6429e886753aaffcbe883. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:02.369452 6426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3f2250a62c6429e886753aaffcbe883. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:02.369864 6293 tablet_service.cc:1467] Processing CreateTablet for tablet e92d6aeeb3df4ea3852c6ebbed0e07c5 (DEFAULT_TABLE table=rebalance_test_table_13 [id=886e8147658c4cffb480ca3fb821c563]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:02.370114 6293 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e92d6aeeb3df4ea3852c6ebbed0e07c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:02.371498 8254 tablet_bootstrap.cc:492] T d3f2250a62c6429e886753aaffcbe883 P 8315d22c56f7439a9e1fc82f153f018c: Bootstrap starting.
I20240501 07:59:02.371678 6555 tablet_service.cc:1467] Processing CreateTablet for tablet 6221deb431d94bfca1c14696e0e0a0c5 (DEFAULT_TABLE table=rebalance_test_table_13 [id=886e8147658c4cffb480ca3fb821c563]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:02.371901 6555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6221deb431d94bfca1c14696e0e0a0c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:02.372236 8256 tablet_bootstrap.cc:492] T e92d6aeeb3df4ea3852c6ebbed0e07c5 P 8b8b3ad3aa8f4053b2167dd550f2086f: Bootstrap starting.
I20240501 07:59:02.372406 6556 tablet_service.cc:1467] Processing CreateTablet for tablet e92d6aeeb3df4ea3852c6ebbed0e07c5 (DEFAULT_TABLE table=rebalance_test_table_13 [id=886e8147658c4cffb480ca3fb821c563]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:02.372603 6556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e92d6aeeb3df4ea3852c6ebbed0e07c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:02.372623 8254 tablet_bootstrap.cc:654] T d3f2250a62c6429e886753aaffcbe883 P 8315d22c56f7439a9e1fc82f153f018c: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:02.373126 8256 tablet_bootstrap.cc:654] T e92d6aeeb3df4ea3852c6ebbed0e07c5 P 8b8b3ad3aa8f4053b2167dd550f2086f: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:02.374995 8254 tablet_bootstrap.cc:492] T d3f2250a62c6429e886753aaffcbe883 P 8315d22c56f7439a9e1fc82f153f018c: No bootstrap required, opened a new log
I20240501 07:59:02.375072 8254 ts_tablet_manager.cc:1367] T d3f2250a62c6429e886753aaffcbe883 P 8315d22c56f7439a9e1fc82f153f018c: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 07:59:02.375201 6425 tablet_service.cc:1467] Processing CreateTablet for tablet e92d6aeeb3df4ea3852c6ebbed0e07c5 (DEFAULT_TABLE table=rebalance_test_table_13 [id=886e8147658c4cffb480ca3fb821c563]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:02.375454 6425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e92d6aeeb3df4ea3852c6ebbed0e07c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:02.375556 8254 raft_consensus.cc:348] T d3f2250a62c6429e886753aaffcbe883 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:59:02.375665 8254 raft_consensus.cc:374] T d3f2250a62c6429e886753aaffcbe883 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:02.375605 6686 tablet_service.cc:1467] Processing CreateTablet for tablet 6221deb431d94bfca1c14696e0e0a0c5 (DEFAULT_TABLE table=rebalance_test_table_13 [id=886e8147658c4cffb480ca3fb821c563]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:02.375695 8254 raft_consensus.cc:729] T d3f2250a62c6429e886753aaffcbe883 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8315d22c56f7439a9e1fc82f153f018c, State: Initialized, Role: FOLLOWER
I20240501 07:59:02.375782 8255 tablet_bootstrap.cc:492] T d3f2250a62c6429e886753aaffcbe883 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Bootstrap starting.
I20240501 07:59:02.375854 6686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6221deb431d94bfca1c14696e0e0a0c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:02.375814 8254 consensus_queue.cc:260] T d3f2250a62c6429e886753aaffcbe883 P 8315d22c56f7439a9e1fc82f153f018c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:59:02.376089 8254 ts_tablet_manager.cc:1398] T d3f2250a62c6429e886753aaffcbe883 P 8315d22c56f7439a9e1fc82f153f018c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:02.376587 8255 tablet_bootstrap.cc:654] T d3f2250a62c6429e886753aaffcbe883 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:02.377043 6426 tablet_service.cc:1467] Processing CreateTablet for tablet 6221deb431d94bfca1c14696e0e0a0c5 (DEFAULT_TABLE table=rebalance_test_table_13 [id=886e8147658c4cffb480ca3fb821c563]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:02.377295 6426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6221deb431d94bfca1c14696e0e0a0c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:02.377507 8254 tablet_bootstrap.cc:492] T e92d6aeeb3df4ea3852c6ebbed0e07c5 P 8315d22c56f7439a9e1fc82f153f018c: Bootstrap starting.
I20240501 07:59:02.377743 6687 tablet_service.cc:1467] Processing CreateTablet for tablet d3f2250a62c6429e886753aaffcbe883 (DEFAULT_TABLE table=rebalance_test_table_13 [id=886e8147658c4cffb480ca3fb821c563]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:02.377972 6687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3f2250a62c6429e886753aaffcbe883. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:02.378397 8254 tablet_bootstrap.cc:654] T e92d6aeeb3df4ea3852c6ebbed0e07c5 P 8315d22c56f7439a9e1fc82f153f018c: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:02.378804 8260 tablet_bootstrap.cc:492] T 6221deb431d94bfca1c14696e0e0a0c5 P 1f9cb72be98941b5a8e504cbe34132b9: Bootstrap starting.
I20240501 07:59:02.379927 8256 tablet_bootstrap.cc:492] T e92d6aeeb3df4ea3852c6ebbed0e07c5 P 8b8b3ad3aa8f4053b2167dd550f2086f: No bootstrap required, opened a new log
I20240501 07:59:02.379962 8260 tablet_bootstrap.cc:654] T 6221deb431d94bfca1c14696e0e0a0c5 P 1f9cb72be98941b5a8e504cbe34132b9: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:02.380091 8255 tablet_bootstrap.cc:492] T d3f2250a62c6429e886753aaffcbe883 P 3b164cbbc00f4e8eb3254ddd349a3ff9: No bootstrap required, opened a new log
I20240501 07:59:02.380106 8256 ts_tablet_manager.cc:1367] T e92d6aeeb3df4ea3852c6ebbed0e07c5 P 8b8b3ad3aa8f4053b2167dd550f2086f: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240501 07:59:02.380612 8256 raft_consensus.cc:348] T e92d6aeeb3df4ea3852c6ebbed0e07c5 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } }
I20240501 07:59:02.380730 8256 raft_consensus.cc:374] T e92d6aeeb3df4ea3852c6ebbed0e07c5 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:02.380767 8256 raft_consensus.cc:729] T e92d6aeeb3df4ea3852c6ebbed0e07c5 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b8b3ad3aa8f4053b2167dd550f2086f, State: Initialized, Role: FOLLOWER
I20240501 07:59:02.380926 8256 consensus_queue.cc:260] T e92d6aeeb3df4ea3852c6ebbed0e07c5 P 8b8b3ad3aa8f4053b2167dd550f2086f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } }
I20240501 07:59:02.381281 8256 ts_tablet_manager.cc:1398] T e92d6aeeb3df4ea3852c6ebbed0e07c5 P 8b8b3ad3aa8f4053b2167dd550f2086f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:02.381354 8254 tablet_bootstrap.cc:492] T e92d6aeeb3df4ea3852c6ebbed0e07c5 P 8315d22c56f7439a9e1fc82f153f018c: No bootstrap required, opened a new log
I20240501 07:59:02.381428 8254 ts_tablet_manager.cc:1367] T e92d6aeeb3df4ea3852c6ebbed0e07c5 P 8315d22c56f7439a9e1fc82f153f018c: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240501 07:59:02.381743 8260 tablet_bootstrap.cc:492] T 6221deb431d94bfca1c14696e0e0a0c5 P 1f9cb72be98941b5a8e504cbe34132b9: No bootstrap required, opened a new log
I20240501 07:59:02.381759 8254 raft_consensus.cc:348] T e92d6aeeb3df4ea3852c6ebbed0e07c5 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } }
I20240501 07:59:02.381832 8260 ts_tablet_manager.cc:1367] T 6221deb431d94bfca1c14696e0e0a0c5 P 1f9cb72be98941b5a8e504cbe34132b9: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 07:59:02.381850 8254 raft_consensus.cc:374] T e92d6aeeb3df4ea3852c6ebbed0e07c5 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:02.381887 8254 raft_consensus.cc:729] T e92d6aeeb3df4ea3852c6ebbed0e07c5 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8315d22c56f7439a9e1fc82f153f018c, State: Initialized, Role: FOLLOWER
I20240501 07:59:02.382019 8254 consensus_queue.cc:260] T e92d6aeeb3df4ea3852c6ebbed0e07c5 P 8315d22c56f7439a9e1fc82f153f018c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } }
I20240501 07:59:02.382295 8254 ts_tablet_manager.cc:1398] T e92d6aeeb3df4ea3852c6ebbed0e07c5 P 8315d22c56f7439a9e1fc82f153f018c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:02.382300 8260 raft_consensus.cc:348] T 6221deb431d94bfca1c14696e0e0a0c5 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:59:02.382395 8260 raft_consensus.cc:374] T 6221deb431d94bfca1c14696e0e0a0c5 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:02.382428 8260 raft_consensus.cc:729] T 6221deb431d94bfca1c14696e0e0a0c5 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f9cb72be98941b5a8e504cbe34132b9, State: Initialized, Role: FOLLOWER
I20240501 07:59:02.382447 8254 tablet_bootstrap.cc:492] T 6221deb431d94bfca1c14696e0e0a0c5 P 8315d22c56f7439a9e1fc82f153f018c: Bootstrap starting.
I20240501 07:59:02.380290 8255 ts_tablet_manager.cc:1367] T d3f2250a62c6429e886753aaffcbe883 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 07:59:02.382550 8260 consensus_queue.cc:260] T 6221deb431d94bfca1c14696e0e0a0c5 P 1f9cb72be98941b5a8e504cbe34132b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:59:02.382813 8260 ts_tablet_manager.cc:1398] T 6221deb431d94bfca1c14696e0e0a0c5 P 1f9cb72be98941b5a8e504cbe34132b9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:02.382990 8260 tablet_bootstrap.cc:492] T d3f2250a62c6429e886753aaffcbe883 P 1f9cb72be98941b5a8e504cbe34132b9: Bootstrap starting.
I20240501 07:59:02.383419 8254 tablet_bootstrap.cc:654] T 6221deb431d94bfca1c14696e0e0a0c5 P 8315d22c56f7439a9e1fc82f153f018c: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:02.383515 8255 raft_consensus.cc:348] T d3f2250a62c6429e886753aaffcbe883 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:59:02.383637 8255 raft_consensus.cc:374] T d3f2250a62c6429e886753aaffcbe883 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:02.383715 8255 raft_consensus.cc:729] T d3f2250a62c6429e886753aaffcbe883 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b164cbbc00f4e8eb3254ddd349a3ff9, State: Initialized, Role: FOLLOWER
I20240501 07:59:02.383792 8260 tablet_bootstrap.cc:654] T d3f2250a62c6429e886753aaffcbe883 P 1f9cb72be98941b5a8e504cbe34132b9: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:02.384101 8255 consensus_queue.cc:260] T d3f2250a62c6429e886753aaffcbe883 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:59:02.384414 8255 ts_tablet_manager.cc:1398] T d3f2250a62c6429e886753aaffcbe883 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:59:02.384596 8255 tablet_bootstrap.cc:492] T 6221deb431d94bfca1c14696e0e0a0c5 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Bootstrap starting.
I20240501 07:59:02.385511 8254 tablet_bootstrap.cc:492] T 6221deb431d94bfca1c14696e0e0a0c5 P 8315d22c56f7439a9e1fc82f153f018c: No bootstrap required, opened a new log
I20240501 07:59:02.385593 8254 ts_tablet_manager.cc:1367] T 6221deb431d94bfca1c14696e0e0a0c5 P 8315d22c56f7439a9e1fc82f153f018c: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240501 07:59:02.385857 8255 tablet_bootstrap.cc:654] T 6221deb431d94bfca1c14696e0e0a0c5 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:02.386087 8254 raft_consensus.cc:348] T 6221deb431d94bfca1c14696e0e0a0c5 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:59:02.386202 8254 raft_consensus.cc:374] T 6221deb431d94bfca1c14696e0e0a0c5 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:02.386250 8254 raft_consensus.cc:729] T 6221deb431d94bfca1c14696e0e0a0c5 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8315d22c56f7439a9e1fc82f153f018c, State: Initialized, Role: FOLLOWER
I20240501 07:59:02.386504 8254 consensus_queue.cc:260] T 6221deb431d94bfca1c14696e0e0a0c5 P 8315d22c56f7439a9e1fc82f153f018c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:59:02.386900 8254 ts_tablet_manager.cc:1398] T 6221deb431d94bfca1c14696e0e0a0c5 P 8315d22c56f7439a9e1fc82f153f018c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:02.387723 8260 tablet_bootstrap.cc:492] T d3f2250a62c6429e886753aaffcbe883 P 1f9cb72be98941b5a8e504cbe34132b9: No bootstrap required, opened a new log
I20240501 07:59:02.387804 8260 ts_tablet_manager.cc:1367] T d3f2250a62c6429e886753aaffcbe883 P 1f9cb72be98941b5a8e504cbe34132b9: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240501 07:59:02.388195 8260 raft_consensus.cc:348] T d3f2250a62c6429e886753aaffcbe883 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:59:02.388291 8260 raft_consensus.cc:374] T d3f2250a62c6429e886753aaffcbe883 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:02.388327 8260 raft_consensus.cc:729] T d3f2250a62c6429e886753aaffcbe883 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f9cb72be98941b5a8e504cbe34132b9, State: Initialized, Role: FOLLOWER
I20240501 07:59:02.388592 8260 consensus_queue.cc:260] T d3f2250a62c6429e886753aaffcbe883 P 1f9cb72be98941b5a8e504cbe34132b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:59:02.388857 8260 ts_tablet_manager.cc:1398] T d3f2250a62c6429e886753aaffcbe883 P 1f9cb72be98941b5a8e504cbe34132b9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:02.389056 8255 tablet_bootstrap.cc:492] T 6221deb431d94bfca1c14696e0e0a0c5 P 3b164cbbc00f4e8eb3254ddd349a3ff9: No bootstrap required, opened a new log
I20240501 07:59:02.389137 8255 ts_tablet_manager.cc:1367] T 6221deb431d94bfca1c14696e0e0a0c5 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 07:59:02.389665 8255 raft_consensus.cc:348] T 6221deb431d94bfca1c14696e0e0a0c5 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:59:02.389772 8255 raft_consensus.cc:374] T 6221deb431d94bfca1c14696e0e0a0c5 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:02.389806 8255 raft_consensus.cc:729] T 6221deb431d94bfca1c14696e0e0a0c5 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b164cbbc00f4e8eb3254ddd349a3ff9, State: Initialized, Role: FOLLOWER
I20240501 07:59:02.389915 8255 consensus_queue.cc:260] T 6221deb431d94bfca1c14696e0e0a0c5 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:59:02.390182 8255 ts_tablet_manager.cc:1398] T 6221deb431d94bfca1c14696e0e0a0c5 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:02.390359 8255 tablet_bootstrap.cc:492] T e92d6aeeb3df4ea3852c6ebbed0e07c5 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Bootstrap starting.
I20240501 07:59:02.391350 8255 tablet_bootstrap.cc:654] T e92d6aeeb3df4ea3852c6ebbed0e07c5 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:02.392786 8255 tablet_bootstrap.cc:492] T e92d6aeeb3df4ea3852c6ebbed0e07c5 P 3b164cbbc00f4e8eb3254ddd349a3ff9: No bootstrap required, opened a new log
I20240501 07:59:02.392869 8255 ts_tablet_manager.cc:1367] T e92d6aeeb3df4ea3852c6ebbed0e07c5 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 07:59:02.393352 8255 raft_consensus.cc:348] T e92d6aeeb3df4ea3852c6ebbed0e07c5 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } }
I20240501 07:59:02.393453 8255 raft_consensus.cc:374] T e92d6aeeb3df4ea3852c6ebbed0e07c5 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:02.393503 8255 raft_consensus.cc:729] T e92d6aeeb3df4ea3852c6ebbed0e07c5 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b164cbbc00f4e8eb3254ddd349a3ff9, State: Initialized, Role: FOLLOWER
I20240501 07:59:02.393621 8255 consensus_queue.cc:260] T e92d6aeeb3df4ea3852c6ebbed0e07c5 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } }
I20240501 07:59:02.393924 8255 ts_tablet_manager.cc:1398] T e92d6aeeb3df4ea3852c6ebbed0e07c5 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:02.426229 8166 raft_consensus.cc:482] T d3f2250a62c6429e886753aaffcbe883 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:02.426347 8166 raft_consensus.cc:504] T d3f2250a62c6429e886753aaffcbe883 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:59:02.426791 8166 leader_election.cc:288] T d3f2250a62c6429e886753aaffcbe883 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130:32941), 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132:36773)
I20240501 07:59:02.427289 6445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3f2250a62c6429e886753aaffcbe883" candidate_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8315d22c56f7439a9e1fc82f153f018c" is_pre_election: true
I20240501 07:59:02.427276 6707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3f2250a62c6429e886753aaffcbe883" candidate_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f9cb72be98941b5a8e504cbe34132b9" is_pre_election: true
I20240501 07:59:02.427434 6445 raft_consensus.cc:2454] T d3f2250a62c6429e886753aaffcbe883 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b164cbbc00f4e8eb3254ddd349a3ff9 in term 0.
I20240501 07:59:02.427449 6707 raft_consensus.cc:2454] T d3f2250a62c6429e886753aaffcbe883 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b164cbbc00f4e8eb3254ddd349a3ff9 in term 0.
I20240501 07:59:02.427784 6512 leader_election.cc:302] T d3f2250a62c6429e886753aaffcbe883 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b164cbbc00f4e8eb3254ddd349a3ff9, 8315d22c56f7439a9e1fc82f153f018c; no voters:
I20240501 07:59:02.427975 8166 raft_consensus.cc:2781] T d3f2250a62c6429e886753aaffcbe883 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:02.428037 8166 raft_consensus.cc:482] T d3f2250a62c6429e886753aaffcbe883 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:02.428092 8166 raft_consensus.cc:3037] T d3f2250a62c6429e886753aaffcbe883 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:02.429235 8166 raft_consensus.cc:504] T d3f2250a62c6429e886753aaffcbe883 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:59:02.429574 8166 leader_election.cc:288] T d3f2250a62c6429e886753aaffcbe883 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [CANDIDATE]: Term 1 election: Requested vote from peers 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130:32941), 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132:36773)
I20240501 07:59:02.429965 6707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3f2250a62c6429e886753aaffcbe883" candidate_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f9cb72be98941b5a8e504cbe34132b9"
I20240501 07:59:02.430069 6707 raft_consensus.cc:3037] T d3f2250a62c6429e886753aaffcbe883 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:02.430358 6445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3f2250a62c6429e886753aaffcbe883" candidate_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8315d22c56f7439a9e1fc82f153f018c"
I20240501 07:59:02.430491 6445 raft_consensus.cc:3037] T d3f2250a62c6429e886753aaffcbe883 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:02.431061 6707 raft_consensus.cc:2454] T d3f2250a62c6429e886753aaffcbe883 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b164cbbc00f4e8eb3254ddd349a3ff9 in term 1.
I20240501 07:59:02.431334 6512 leader_election.cc:302] T d3f2250a62c6429e886753aaffcbe883 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f9cb72be98941b5a8e504cbe34132b9, 3b164cbbc00f4e8eb3254ddd349a3ff9; no voters:
I20240501 07:59:02.431501 6445 raft_consensus.cc:2454] T d3f2250a62c6429e886753aaffcbe883 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b164cbbc00f4e8eb3254ddd349a3ff9 in term 1.
I20240501 07:59:02.431542 8166 raft_consensus.cc:2781] T d3f2250a62c6429e886753aaffcbe883 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:02.431687 8166 raft_consensus.cc:686] T d3f2250a62c6429e886753aaffcbe883 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 LEADER]: Becoming Leader. State: Replica: 3b164cbbc00f4e8eb3254ddd349a3ff9, State: Running, Role: LEADER
I20240501 07:59:02.431835 8166 consensus_queue.cc:237] T d3f2250a62c6429e886753aaffcbe883 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:59:02.433835 6169 catalog_manager.cc:5484] T d3f2250a62c6429e886753aaffcbe883 P 3b164cbbc00f4e8eb3254ddd349a3ff9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } health_report { overall_health: UNKNOWN } } }
I20240501 07:59:02.454591 8253 raft_consensus.cc:482] T e92d6aeeb3df4ea3852c6ebbed0e07c5 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:02.454735 8253 raft_consensus.cc:504] T e92d6aeeb3df4ea3852c6ebbed0e07c5 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } }
I20240501 07:59:02.455139 8253 leader_election.cc:288] T e92d6aeeb3df4ea3852c6ebbed0e07c5 P 8315d22c56f7439a9e1fc82f153f018c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131:34471), 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129:46875)
I20240501 07:59:02.455593 6577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e92d6aeeb3df4ea3852c6ebbed0e07c5" candidate_uuid: "8315d22c56f7439a9e1fc82f153f018c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" is_pre_election: true
I20240501 07:59:02.455732 6577 raft_consensus.cc:2454] T e92d6aeeb3df4ea3852c6ebbed0e07c5 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8315d22c56f7439a9e1fc82f153f018c in term 0.
I20240501 07:59:02.455914 6313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e92d6aeeb3df4ea3852c6ebbed0e07c5" candidate_uuid: "8315d22c56f7439a9e1fc82f153f018c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" is_pre_election: true
I20240501 07:59:02.456007 6382 leader_election.cc:302] T e92d6aeeb3df4ea3852c6ebbed0e07c5 P 8315d22c56f7439a9e1fc82f153f018c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b164cbbc00f4e8eb3254ddd349a3ff9, 8315d22c56f7439a9e1fc82f153f018c; no voters:
I20240501 07:59:02.456040 6313 raft_consensus.cc:2454] T e92d6aeeb3df4ea3852c6ebbed0e07c5 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8315d22c56f7439a9e1fc82f153f018c in term 0.
I20240501 07:59:02.456238 8253 raft_consensus.cc:2781] T e92d6aeeb3df4ea3852c6ebbed0e07c5 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:02.456302 8253 raft_consensus.cc:482] T e92d6aeeb3df4ea3852c6ebbed0e07c5 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:02.456362 8253 raft_consensus.cc:3037] T e92d6aeeb3df4ea3852c6ebbed0e07c5 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:02.457285 8253 raft_consensus.cc:504] T e92d6aeeb3df4ea3852c6ebbed0e07c5 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } }
I20240501 07:59:02.457604 8253 leader_election.cc:288] T e92d6aeeb3df4ea3852c6ebbed0e07c5 P 8315d22c56f7439a9e1fc82f153f018c [CANDIDATE]: Term 1 election: Requested vote from peers 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131:34471), 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129:46875)
I20240501 07:59:02.457937 6577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e92d6aeeb3df4ea3852c6ebbed0e07c5" candidate_uuid: "8315d22c56f7439a9e1fc82f153f018c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9"
I20240501 07:59:02.457973 6313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e92d6aeeb3df4ea3852c6ebbed0e07c5" candidate_uuid: "8315d22c56f7439a9e1fc82f153f018c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f"
I20240501 07:59:02.458065 6577 raft_consensus.cc:3037] T e92d6aeeb3df4ea3852c6ebbed0e07c5 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:02.458094 6313 raft_consensus.cc:3037] T e92d6aeeb3df4ea3852c6ebbed0e07c5 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:02.458885 6313 raft_consensus.cc:2454] T e92d6aeeb3df4ea3852c6ebbed0e07c5 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8315d22c56f7439a9e1fc82f153f018c in term 1.
I20240501 07:59:02.458994 6577 raft_consensus.cc:2454] T e92d6aeeb3df4ea3852c6ebbed0e07c5 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8315d22c56f7439a9e1fc82f153f018c in term 1.
I20240501 07:59:02.459223 6379 leader_election.cc:302] T e92d6aeeb3df4ea3852c6ebbed0e07c5 P 8315d22c56f7439a9e1fc82f153f018c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8315d22c56f7439a9e1fc82f153f018c, 8b8b3ad3aa8f4053b2167dd550f2086f; no voters:
I20240501 07:59:02.459369 8253 raft_consensus.cc:2781] T e92d6aeeb3df4ea3852c6ebbed0e07c5 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:02.459453 8253 raft_consensus.cc:686] T e92d6aeeb3df4ea3852c6ebbed0e07c5 P 8315d22c56f7439a9e1fc82f153f018c [term 1 LEADER]: Becoming Leader. State: Replica: 8315d22c56f7439a9e1fc82f153f018c, State: Running, Role: LEADER
I20240501 07:59:02.459600 8253 consensus_queue.cc:237] T e92d6aeeb3df4ea3852c6ebbed0e07c5 P 8315d22c56f7439a9e1fc82f153f018c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } }
I20240501 07:59:02.461544 6169 catalog_manager.cc:5484] T e92d6aeeb3df4ea3852c6ebbed0e07c5 P 8315d22c56f7439a9e1fc82f153f018c reported cstate change: term changed from 0 to 1, leader changed from <none> to 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130). New cstate: current_term: 1 leader_uuid: "8315d22c56f7439a9e1fc82f153f018c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } health_report { overall_health: HEALTHY } } }
I20240501 07:59:02.709106 8223 raft_consensus.cc:482] T 6221deb431d94bfca1c14696e0e0a0c5 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:02.709235 8223 raft_consensus.cc:504] T 6221deb431d94bfca1c14696e0e0a0c5 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:59:02.709645 8223 leader_election.cc:288] T 6221deb431d94bfca1c14696e0e0a0c5 P 1f9cb72be98941b5a8e504cbe34132b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131:34471), 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130:32941)
I20240501 07:59:02.710024 6445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6221deb431d94bfca1c14696e0e0a0c5" candidate_uuid: "1f9cb72be98941b5a8e504cbe34132b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8315d22c56f7439a9e1fc82f153f018c" is_pre_election: true
I20240501 07:59:02.710026 6577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6221deb431d94bfca1c14696e0e0a0c5" candidate_uuid: "1f9cb72be98941b5a8e504cbe34132b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" is_pre_election: true
I20240501 07:59:02.710156 6577 raft_consensus.cc:2454] T 6221deb431d94bfca1c14696e0e0a0c5 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f9cb72be98941b5a8e504cbe34132b9 in term 0.
I20240501 07:59:02.710166 6445 raft_consensus.cc:2454] T 6221deb431d94bfca1c14696e0e0a0c5 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f9cb72be98941b5a8e504cbe34132b9 in term 0.
I20240501 07:59:02.710453 6643 leader_election.cc:302] T 6221deb431d94bfca1c14696e0e0a0c5 P 1f9cb72be98941b5a8e504cbe34132b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f9cb72be98941b5a8e504cbe34132b9, 8315d22c56f7439a9e1fc82f153f018c; no voters:
I20240501 07:59:02.710608 8223 raft_consensus.cc:2781] T 6221deb431d94bfca1c14696e0e0a0c5 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:02.710671 8223 raft_consensus.cc:482] T 6221deb431d94bfca1c14696e0e0a0c5 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:02.710704 8223 raft_consensus.cc:3037] T 6221deb431d94bfca1c14696e0e0a0c5 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:02.711532 8223 raft_consensus.cc:504] T 6221deb431d94bfca1c14696e0e0a0c5 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:59:02.711809 8223 leader_election.cc:288] T 6221deb431d94bfca1c14696e0e0a0c5 P 1f9cb72be98941b5a8e504cbe34132b9 [CANDIDATE]: Term 1 election: Requested vote from peers 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131:34471), 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130:32941)
I20240501 07:59:02.712119 6577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6221deb431d94bfca1c14696e0e0a0c5" candidate_uuid: "1f9cb72be98941b5a8e504cbe34132b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9"
I20240501 07:59:02.712164 6445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6221deb431d94bfca1c14696e0e0a0c5" candidate_uuid: "1f9cb72be98941b5a8e504cbe34132b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8315d22c56f7439a9e1fc82f153f018c"
I20240501 07:59:02.712262 6445 raft_consensus.cc:3037] T 6221deb431d94bfca1c14696e0e0a0c5 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:02.712278 6577 raft_consensus.cc:3037] T 6221deb431d94bfca1c14696e0e0a0c5 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:02.713017 6445 raft_consensus.cc:2454] T 6221deb431d94bfca1c14696e0e0a0c5 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f9cb72be98941b5a8e504cbe34132b9 in term 1.
I20240501 07:59:02.713039 6577 raft_consensus.cc:2454] T 6221deb431d94bfca1c14696e0e0a0c5 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f9cb72be98941b5a8e504cbe34132b9 in term 1.
I20240501 07:59:02.713300 6644 leader_election.cc:302] T 6221deb431d94bfca1c14696e0e0a0c5 P 1f9cb72be98941b5a8e504cbe34132b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f9cb72be98941b5a8e504cbe34132b9, 3b164cbbc00f4e8eb3254ddd349a3ff9; no voters:
I20240501 07:59:02.713456 8223 raft_consensus.cc:2781] T 6221deb431d94bfca1c14696e0e0a0c5 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:02.713543 8223 raft_consensus.cc:686] T 6221deb431d94bfca1c14696e0e0a0c5 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 LEADER]: Becoming Leader. State: Replica: 1f9cb72be98941b5a8e504cbe34132b9, State: Running, Role: LEADER
I20240501 07:59:02.713681 8223 consensus_queue.cc:237] T 6221deb431d94bfca1c14696e0e0a0c5 P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:59:02.715608 6169 catalog_manager.cc:5484] T 6221deb431d94bfca1c14696e0e0a0c5 P 1f9cb72be98941b5a8e504cbe34132b9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132). New cstate: current_term: 1 leader_uuid: "1f9cb72be98941b5a8e504cbe34132b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } health_report { overall_health: HEALTHY } } }
W20240501 07:59:02.858394 8270 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:59:02.858646 8270 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:59:02.861713 8270 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_13'
I20240501 07:59:02.885660 8235 consensus_queue.cc:1035] T d3f2250a62c6429e886753aaffcbe883 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:02.891455 8166 consensus_queue.cc:1035] T d3f2250a62c6429e886753aaffcbe883 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:02.919193 8193 consensus_queue.cc:1035] T e92d6aeeb3df4ea3852c6ebbed0e07c5 P 8315d22c56f7439a9e1fc82f153f018c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:02.924523 8253 consensus_queue.cc:1035] T e92d6aeeb3df4ea3852c6ebbed0e07c5 P 8315d22c56f7439a9e1fc82f153f018c [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:02.969120 6577 raft_consensus.cc:1261] T 6221deb431d94bfca1c14696e0e0a0c5 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Refusing update from remote peer 1f9cb72be98941b5a8e504cbe34132b9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:02.969640 8223 consensus_queue.cc:1035] T 6221deb431d94bfca1c14696e0e0a0c5 P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:02.971336 6445 raft_consensus.cc:1261] T 6221deb431d94bfca1c14696e0e0a0c5 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Refusing update from remote peer 1f9cb72be98941b5a8e504cbe34132b9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:02.971849 8223 consensus_queue.cc:1035] T 6221deb431d94bfca1c14696e0e0a0c5 P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:02.985508 8287 mvcc.cc:205] Tried to move back new op lower bound from 7022798204740427776 to 7022798203756544000. Current Snapshot: MvccSnapshot[applied={T|T < 7022798204740427776}]
INSERT report
rows total: 2000
time total: 123.287 ms
time per row: 0.0616435 ms
I20240501 07:59:03.037387 6169 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55950:
name: "rebalance_test_table_14"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 07:59:03.044270 6687 tablet_service.cc:1467] Processing CreateTablet for tablet 8f56a28665f342c5a77baf86dbe7d041 (DEFAULT_TABLE table=rebalance_test_table_14 [id=6a104310158f4500bd3056343aa6eeca]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:03.044407 6426 tablet_service.cc:1467] Processing CreateTablet for tablet 8f56a28665f342c5a77baf86dbe7d041 (DEFAULT_TABLE table=rebalance_test_table_14 [id=6a104310158f4500bd3056343aa6eeca]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:03.044610 6687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f56a28665f342c5a77baf86dbe7d041. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:03.044693 6426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f56a28665f342c5a77baf86dbe7d041. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:03.045159 6425 tablet_service.cc:1467] Processing CreateTablet for tablet b914638d3a884ae4932316bdbd1fb4a4 (DEFAULT_TABLE table=rebalance_test_table_14 [id=6a104310158f4500bd3056343aa6eeca]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:03.045360 6425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b914638d3a884ae4932316bdbd1fb4a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:03.046851 8331 tablet_bootstrap.cc:492] T 8f56a28665f342c5a77baf86dbe7d041 P 8315d22c56f7439a9e1fc82f153f018c: Bootstrap starting.
I20240501 07:59:03.047194 6556 tablet_service.cc:1467] Processing CreateTablet for tablet 3c0ec93103d349b5b57853f4b4e9f8db (DEFAULT_TABLE table=rebalance_test_table_14 [id=6a104310158f4500bd3056343aa6eeca]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:03.047513 6556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c0ec93103d349b5b57853f4b4e9f8db. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:03.047452 6294 tablet_service.cc:1467] Processing CreateTablet for tablet 3c0ec93103d349b5b57853f4b4e9f8db (DEFAULT_TABLE table=rebalance_test_table_14 [id=6a104310158f4500bd3056343aa6eeca]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:03.047715 6294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c0ec93103d349b5b57853f4b4e9f8db. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:03.047888 8331 tablet_bootstrap.cc:654] T 8f56a28665f342c5a77baf86dbe7d041 P 8315d22c56f7439a9e1fc82f153f018c: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:03.049041 8331 tablet_bootstrap.cc:492] T 8f56a28665f342c5a77baf86dbe7d041 P 8315d22c56f7439a9e1fc82f153f018c: No bootstrap required, opened a new log
I20240501 07:59:03.049120 8331 ts_tablet_manager.cc:1367] T 8f56a28665f342c5a77baf86dbe7d041 P 8315d22c56f7439a9e1fc82f153f018c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 07:59:03.049703 8334 tablet_bootstrap.cc:492] T 3c0ec93103d349b5b57853f4b4e9f8db P 3b164cbbc00f4e8eb3254ddd349a3ff9: Bootstrap starting.
I20240501 07:59:03.049594 8331 raft_consensus.cc:348] T 8f56a28665f342c5a77baf86dbe7d041 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } }
I20240501 07:59:03.049698 6293 tablet_service.cc:1467] Processing CreateTablet for tablet 8f56a28665f342c5a77baf86dbe7d041 (DEFAULT_TABLE table=rebalance_test_table_14 [id=6a104310158f4500bd3056343aa6eeca]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:03.049789 8331 raft_consensus.cc:374] T 8f56a28665f342c5a77baf86dbe7d041 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:03.049849 8331 raft_consensus.cc:729] T 8f56a28665f342c5a77baf86dbe7d041 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8315d22c56f7439a9e1fc82f153f018c, State: Initialized, Role: FOLLOWER
I20240501 07:59:03.049934 6293 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f56a28665f342c5a77baf86dbe7d041. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:03.049974 8331 consensus_queue.cc:260] T 8f56a28665f342c5a77baf86dbe7d041 P 8315d22c56f7439a9e1fc82f153f018c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } }
I20240501 07:59:03.050184 6295 tablet_service.cc:1467] Processing CreateTablet for tablet b914638d3a884ae4932316bdbd1fb4a4 (DEFAULT_TABLE table=rebalance_test_table_14 [id=6a104310158f4500bd3056343aa6eeca]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:03.050388 6295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b914638d3a884ae4932316bdbd1fb4a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:03.050632 8334 tablet_bootstrap.cc:654] T 3c0ec93103d349b5b57853f4b4e9f8db P 3b164cbbc00f4e8eb3254ddd349a3ff9: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:03.050915 8331 ts_tablet_manager.cc:1398] T 8f56a28665f342c5a77baf86dbe7d041 P 8315d22c56f7439a9e1fc82f153f018c: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:59:03.051065 8331 tablet_bootstrap.cc:492] T b914638d3a884ae4932316bdbd1fb4a4 P 8315d22c56f7439a9e1fc82f153f018c: Bootstrap starting.
I20240501 07:59:03.051927 8331 tablet_bootstrap.cc:654] T b914638d3a884ae4932316bdbd1fb4a4 P 8315d22c56f7439a9e1fc82f153f018c: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:03.052763 6687 tablet_service.cc:1467] Processing CreateTablet for tablet 3c0ec93103d349b5b57853f4b4e9f8db (DEFAULT_TABLE table=rebalance_test_table_14 [id=6a104310158f4500bd3056343aa6eeca]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:03.052786 6686 tablet_service.cc:1467] Processing CreateTablet for tablet b914638d3a884ae4932316bdbd1fb4a4 (DEFAULT_TABLE table=rebalance_test_table_14 [id=6a104310158f4500bd3056343aa6eeca]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:03.053229 6686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b914638d3a884ae4932316bdbd1fb4a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:03.053318 8332 tablet_bootstrap.cc:492] T 8f56a28665f342c5a77baf86dbe7d041 P 1f9cb72be98941b5a8e504cbe34132b9: Bootstrap starting.
I20240501 07:59:03.053992 6687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c0ec93103d349b5b57853f4b4e9f8db. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:03.054177 8332 tablet_bootstrap.cc:654] T 8f56a28665f342c5a77baf86dbe7d041 P 1f9cb72be98941b5a8e504cbe34132b9: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:03.055284 8331 tablet_bootstrap.cc:492] T b914638d3a884ae4932316bdbd1fb4a4 P 8315d22c56f7439a9e1fc82f153f018c: No bootstrap required, opened a new log
I20240501 07:59:03.055449 8331 ts_tablet_manager.cc:1367] T b914638d3a884ae4932316bdbd1fb4a4 P 8315d22c56f7439a9e1fc82f153f018c: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 07:59:03.056061 8335 tablet_bootstrap.cc:492] T 3c0ec93103d349b5b57853f4b4e9f8db P 8b8b3ad3aa8f4053b2167dd550f2086f: Bootstrap starting.
I20240501 07:59:03.056047 8331 raft_consensus.cc:348] T b914638d3a884ae4932316bdbd1fb4a4 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } }
I20240501 07:59:03.056201 8331 raft_consensus.cc:374] T b914638d3a884ae4932316bdbd1fb4a4 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:03.056252 8331 raft_consensus.cc:729] T b914638d3a884ae4932316bdbd1fb4a4 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8315d22c56f7439a9e1fc82f153f018c, State: Initialized, Role: FOLLOWER
I20240501 07:59:03.056378 8331 consensus_queue.cc:260] T b914638d3a884ae4932316bdbd1fb4a4 P 8315d22c56f7439a9e1fc82f153f018c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } }
I20240501 07:59:03.056667 8331 ts_tablet_manager.cc:1398] T b914638d3a884ae4932316bdbd1fb4a4 P 8315d22c56f7439a9e1fc82f153f018c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:03.057682 8334 tablet_bootstrap.cc:492] T 3c0ec93103d349b5b57853f4b4e9f8db P 3b164cbbc00f4e8eb3254ddd349a3ff9: No bootstrap required, opened a new log
I20240501 07:59:03.057783 8335 tablet_bootstrap.cc:654] T 3c0ec93103d349b5b57853f4b4e9f8db P 8b8b3ad3aa8f4053b2167dd550f2086f: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:03.057790 8334 ts_tablet_manager.cc:1367] T 3c0ec93103d349b5b57853f4b4e9f8db P 3b164cbbc00f4e8eb3254ddd349a3ff9: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240501 07:59:03.058279 8334 raft_consensus.cc:348] T 3c0ec93103d349b5b57853f4b4e9f8db P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } }
I20240501 07:59:03.058462 8334 raft_consensus.cc:374] T 3c0ec93103d349b5b57853f4b4e9f8db P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:03.058521 8334 raft_consensus.cc:729] T 3c0ec93103d349b5b57853f4b4e9f8db P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b164cbbc00f4e8eb3254ddd349a3ff9, State: Initialized, Role: FOLLOWER
I20240501 07:59:03.059195 8334 consensus_queue.cc:260] T 3c0ec93103d349b5b57853f4b4e9f8db P 3b164cbbc00f4e8eb3254ddd349a3ff9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } }
I20240501 07:59:03.059958 8332 tablet_bootstrap.cc:492] T 8f56a28665f342c5a77baf86dbe7d041 P 1f9cb72be98941b5a8e504cbe34132b9: No bootstrap required, opened a new log
I20240501 07:59:03.060240 8332 ts_tablet_manager.cc:1367] T 8f56a28665f342c5a77baf86dbe7d041 P 1f9cb72be98941b5a8e504cbe34132b9: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240501 07:59:03.060401 8334 ts_tablet_manager.cc:1398] T 3c0ec93103d349b5b57853f4b4e9f8db P 3b164cbbc00f4e8eb3254ddd349a3ff9: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 07:59:03.060088 8335 tablet_bootstrap.cc:492] T 3c0ec93103d349b5b57853f4b4e9f8db P 8b8b3ad3aa8f4053b2167dd550f2086f: No bootstrap required, opened a new log
I20240501 07:59:03.060564 8335 ts_tablet_manager.cc:1367] T 3c0ec93103d349b5b57853f4b4e9f8db P 8b8b3ad3aa8f4053b2167dd550f2086f: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20240501 07:59:03.060846 8332 raft_consensus.cc:348] T 8f56a28665f342c5a77baf86dbe7d041 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } }
I20240501 07:59:03.060945 8332 raft_consensus.cc:374] T 8f56a28665f342c5a77baf86dbe7d041 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:03.061017 8332 raft_consensus.cc:729] T 8f56a28665f342c5a77baf86dbe7d041 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f9cb72be98941b5a8e504cbe34132b9, State: Initialized, Role: FOLLOWER
I20240501 07:59:03.061038 8335 raft_consensus.cc:348] T 3c0ec93103d349b5b57853f4b4e9f8db P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } }
I20240501 07:59:03.061225 8332 consensus_queue.cc:260] T 8f56a28665f342c5a77baf86dbe7d041 P 1f9cb72be98941b5a8e504cbe34132b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } }
I20240501 07:59:03.061195 8335 raft_consensus.cc:374] T 3c0ec93103d349b5b57853f4b4e9f8db P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:03.061420 8335 raft_consensus.cc:729] T 3c0ec93103d349b5b57853f4b4e9f8db P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b8b3ad3aa8f4053b2167dd550f2086f, State: Initialized, Role: FOLLOWER
I20240501 07:59:03.061595 8332 ts_tablet_manager.cc:1398] T 8f56a28665f342c5a77baf86dbe7d041 P 1f9cb72be98941b5a8e504cbe34132b9: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 07:59:03.061554 8335 consensus_queue.cc:260] T 3c0ec93103d349b5b57853f4b4e9f8db P 8b8b3ad3aa8f4053b2167dd550f2086f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } }
I20240501 07:59:03.061770 8332 tablet_bootstrap.cc:492] T b914638d3a884ae4932316bdbd1fb4a4 P 1f9cb72be98941b5a8e504cbe34132b9: Bootstrap starting.
I20240501 07:59:03.062386 8335 ts_tablet_manager.cc:1398] T 3c0ec93103d349b5b57853f4b4e9f8db P 8b8b3ad3aa8f4053b2167dd550f2086f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:59:03.062561 8335 tablet_bootstrap.cc:492] T 8f56a28665f342c5a77baf86dbe7d041 P 8b8b3ad3aa8f4053b2167dd550f2086f: Bootstrap starting.
I20240501 07:59:03.063150 8332 tablet_bootstrap.cc:654] T b914638d3a884ae4932316bdbd1fb4a4 P 1f9cb72be98941b5a8e504cbe34132b9: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:03.063478 8335 tablet_bootstrap.cc:654] T 8f56a28665f342c5a77baf86dbe7d041 P 8b8b3ad3aa8f4053b2167dd550f2086f: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:03.064574 8332 tablet_bootstrap.cc:492] T b914638d3a884ae4932316bdbd1fb4a4 P 1f9cb72be98941b5a8e504cbe34132b9: No bootstrap required, opened a new log
I20240501 07:59:03.064661 8332 ts_tablet_manager.cc:1367] T b914638d3a884ae4932316bdbd1fb4a4 P 1f9cb72be98941b5a8e504cbe34132b9: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240501 07:59:03.065099 8332 raft_consensus.cc:348] T b914638d3a884ae4932316bdbd1fb4a4 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } }
I20240501 07:59:03.065205 8332 raft_consensus.cc:374] T b914638d3a884ae4932316bdbd1fb4a4 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:03.065235 8332 raft_consensus.cc:729] T b914638d3a884ae4932316bdbd1fb4a4 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f9cb72be98941b5a8e504cbe34132b9, State: Initialized, Role: FOLLOWER
I20240501 07:59:03.065393 8335 tablet_bootstrap.cc:492] T 8f56a28665f342c5a77baf86dbe7d041 P 8b8b3ad3aa8f4053b2167dd550f2086f: No bootstrap required, opened a new log
I20240501 07:59:03.065379 8332 consensus_queue.cc:260] T b914638d3a884ae4932316bdbd1fb4a4 P 1f9cb72be98941b5a8e504cbe34132b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } }
I20240501 07:59:03.065475 8335 ts_tablet_manager.cc:1367] T 8f56a28665f342c5a77baf86dbe7d041 P 8b8b3ad3aa8f4053b2167dd550f2086f: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240501 07:59:03.065651 8332 ts_tablet_manager.cc:1398] T b914638d3a884ae4932316bdbd1fb4a4 P 1f9cb72be98941b5a8e504cbe34132b9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:03.065840 8332 tablet_bootstrap.cc:492] T 3c0ec93103d349b5b57853f4b4e9f8db P 1f9cb72be98941b5a8e504cbe34132b9: Bootstrap starting.
I20240501 07:59:03.065930 8335 raft_consensus.cc:348] T 8f56a28665f342c5a77baf86dbe7d041 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } }
I20240501 07:59:03.066035 8335 raft_consensus.cc:374] T 8f56a28665f342c5a77baf86dbe7d041 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:03.066076 8335 raft_consensus.cc:729] T 8f56a28665f342c5a77baf86dbe7d041 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b8b3ad3aa8f4053b2167dd550f2086f, State: Initialized, Role: FOLLOWER
I20240501 07:59:03.066195 8335 consensus_queue.cc:260] T 8f56a28665f342c5a77baf86dbe7d041 P 8b8b3ad3aa8f4053b2167dd550f2086f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } }
I20240501 07:59:03.066453 8335 ts_tablet_manager.cc:1398] T 8f56a28665f342c5a77baf86dbe7d041 P 8b8b3ad3aa8f4053b2167dd550f2086f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:59:03.066622 8335 tablet_bootstrap.cc:492] T b914638d3a884ae4932316bdbd1fb4a4 P 8b8b3ad3aa8f4053b2167dd550f2086f: Bootstrap starting.
I20240501 07:59:03.066867 8332 tablet_bootstrap.cc:654] T 3c0ec93103d349b5b57853f4b4e9f8db P 1f9cb72be98941b5a8e504cbe34132b9: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:03.067495 8335 tablet_bootstrap.cc:654] T b914638d3a884ae4932316bdbd1fb4a4 P 8b8b3ad3aa8f4053b2167dd550f2086f: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:03.068265 8332 tablet_bootstrap.cc:492] T 3c0ec93103d349b5b57853f4b4e9f8db P 1f9cb72be98941b5a8e504cbe34132b9: No bootstrap required, opened a new log
I20240501 07:59:03.068344 8332 ts_tablet_manager.cc:1367] T 3c0ec93103d349b5b57853f4b4e9f8db P 1f9cb72be98941b5a8e504cbe34132b9: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20240501 07:59:03.068640 8335 tablet_bootstrap.cc:492] T b914638d3a884ae4932316bdbd1fb4a4 P 8b8b3ad3aa8f4053b2167dd550f2086f: No bootstrap required, opened a new log
I20240501 07:59:03.068727 8335 ts_tablet_manager.cc:1367] T b914638d3a884ae4932316bdbd1fb4a4 P 8b8b3ad3aa8f4053b2167dd550f2086f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240501 07:59:03.068711 8332 raft_consensus.cc:348] T 3c0ec93103d349b5b57853f4b4e9f8db P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } }
I20240501 07:59:03.068816 8332 raft_consensus.cc:374] T 3c0ec93103d349b5b57853f4b4e9f8db P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:03.068854 8332 raft_consensus.cc:729] T 3c0ec93103d349b5b57853f4b4e9f8db P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f9cb72be98941b5a8e504cbe34132b9, State: Initialized, Role: FOLLOWER
I20240501 07:59:03.069075 8332 consensus_queue.cc:260] T 3c0ec93103d349b5b57853f4b4e9f8db P 1f9cb72be98941b5a8e504cbe34132b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } }
I20240501 07:59:03.069377 8335 raft_consensus.cc:348] T b914638d3a884ae4932316bdbd1fb4a4 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } }
I20240501 07:59:03.069495 8335 raft_consensus.cc:374] T b914638d3a884ae4932316bdbd1fb4a4 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:03.069547 8335 raft_consensus.cc:729] T b914638d3a884ae4932316bdbd1fb4a4 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b8b3ad3aa8f4053b2167dd550f2086f, State: Initialized, Role: FOLLOWER
I20240501 07:59:03.069609 8332 ts_tablet_manager.cc:1398] T 3c0ec93103d349b5b57853f4b4e9f8db P 1f9cb72be98941b5a8e504cbe34132b9: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240501 07:59:03.069682 8335 consensus_queue.cc:260] T b914638d3a884ae4932316bdbd1fb4a4 P 8b8b3ad3aa8f4053b2167dd550f2086f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } }
I20240501 07:59:03.069970 8335 ts_tablet_manager.cc:1398] T b914638d3a884ae4932316bdbd1fb4a4 P 8b8b3ad3aa8f4053b2167dd550f2086f: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 07:59:03.100514 8166 raft_consensus.cc:482] T 3c0ec93103d349b5b57853f4b4e9f8db P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:03.100653 8166 raft_consensus.cc:504] T 3c0ec93103d349b5b57853f4b4e9f8db P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } }
I20240501 07:59:03.101076 8166 leader_election.cc:288] T 3c0ec93103d349b5b57853f4b4e9f8db P 3b164cbbc00f4e8eb3254ddd349a3ff9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129:46875), 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132:36773)
I20240501 07:59:03.101490 6708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c0ec93103d349b5b57853f4b4e9f8db" candidate_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f9cb72be98941b5a8e504cbe34132b9" is_pre_election: true
I20240501 07:59:03.101465 6313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c0ec93103d349b5b57853f4b4e9f8db" candidate_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" is_pre_election: true
I20240501 07:59:03.101625 6313 raft_consensus.cc:2454] T 3c0ec93103d349b5b57853f4b4e9f8db P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b164cbbc00f4e8eb3254ddd349a3ff9 in term 0.
I20240501 07:59:03.101625 6708 raft_consensus.cc:2454] T 3c0ec93103d349b5b57853f4b4e9f8db P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b164cbbc00f4e8eb3254ddd349a3ff9 in term 0.
I20240501 07:59:03.101922 6512 leader_election.cc:302] T 3c0ec93103d349b5b57853f4b4e9f8db P 3b164cbbc00f4e8eb3254ddd349a3ff9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f9cb72be98941b5a8e504cbe34132b9, 3b164cbbc00f4e8eb3254ddd349a3ff9; no voters:
I20240501 07:59:03.102093 8166 raft_consensus.cc:2781] T 3c0ec93103d349b5b57853f4b4e9f8db P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:03.102157 8166 raft_consensus.cc:482] T 3c0ec93103d349b5b57853f4b4e9f8db P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:03.102186 8166 raft_consensus.cc:3037] T 3c0ec93103d349b5b57853f4b4e9f8db P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:03.102988 8166 raft_consensus.cc:504] T 3c0ec93103d349b5b57853f4b4e9f8db P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } }
I20240501 07:59:03.103308 8166 leader_election.cc:288] T 3c0ec93103d349b5b57853f4b4e9f8db P 3b164cbbc00f4e8eb3254ddd349a3ff9 [CANDIDATE]: Term 1 election: Requested vote from peers 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129:46875), 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132:36773)
I20240501 07:59:03.103704 6708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c0ec93103d349b5b57853f4b4e9f8db" candidate_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f9cb72be98941b5a8e504cbe34132b9"
I20240501 07:59:03.103718 6313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c0ec93103d349b5b57853f4b4e9f8db" candidate_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f"
I20240501 07:59:03.103830 6708 raft_consensus.cc:3037] T 3c0ec93103d349b5b57853f4b4e9f8db P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:03.103845 6313 raft_consensus.cc:3037] T 3c0ec93103d349b5b57853f4b4e9f8db P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:03.104802 6708 raft_consensus.cc:2454] T 3c0ec93103d349b5b57853f4b4e9f8db P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b164cbbc00f4e8eb3254ddd349a3ff9 in term 1.
I20240501 07:59:03.104802 6313 raft_consensus.cc:2454] T 3c0ec93103d349b5b57853f4b4e9f8db P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b164cbbc00f4e8eb3254ddd349a3ff9 in term 1.
I20240501 07:59:03.105131 6512 leader_election.cc:302] T 3c0ec93103d349b5b57853f4b4e9f8db P 3b164cbbc00f4e8eb3254ddd349a3ff9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f9cb72be98941b5a8e504cbe34132b9, 3b164cbbc00f4e8eb3254ddd349a3ff9; no voters:
I20240501 07:59:03.105402 8166 raft_consensus.cc:2781] T 3c0ec93103d349b5b57853f4b4e9f8db P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:03.105512 8166 raft_consensus.cc:686] T 3c0ec93103d349b5b57853f4b4e9f8db P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 LEADER]: Becoming Leader. State: Replica: 3b164cbbc00f4e8eb3254ddd349a3ff9, State: Running, Role: LEADER
I20240501 07:59:03.105656 8166 consensus_queue.cc:237] T 3c0ec93103d349b5b57853f4b4e9f8db P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } }
I20240501 07:59:03.107596 6172 catalog_manager.cc:5484] T 3c0ec93103d349b5b57853f4b4e9f8db P 3b164cbbc00f4e8eb3254ddd349a3ff9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } health_report { overall_health: HEALTHY } } }
I20240501 07:59:03.123591 8193 raft_consensus.cc:482] T 8f56a28665f342c5a77baf86dbe7d041 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:03.123704 8193 raft_consensus.cc:504] T 8f56a28665f342c5a77baf86dbe7d041 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } }
I20240501 07:59:03.124099 8193 leader_election.cc:288] T 8f56a28665f342c5a77baf86dbe7d041 P 8315d22c56f7439a9e1fc82f153f018c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132:36773), 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129:46875)
I20240501 07:59:03.124384 6708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f56a28665f342c5a77baf86dbe7d041" candidate_uuid: "8315d22c56f7439a9e1fc82f153f018c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f9cb72be98941b5a8e504cbe34132b9" is_pre_election: true
I20240501 07:59:03.124441 6313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f56a28665f342c5a77baf86dbe7d041" candidate_uuid: "8315d22c56f7439a9e1fc82f153f018c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" is_pre_election: true
I20240501 07:59:03.124512 6708 raft_consensus.cc:2454] T 8f56a28665f342c5a77baf86dbe7d041 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8315d22c56f7439a9e1fc82f153f018c in term 0.
I20240501 07:59:03.124580 6313 raft_consensus.cc:2454] T 8f56a28665f342c5a77baf86dbe7d041 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8315d22c56f7439a9e1fc82f153f018c in term 0.
I20240501 07:59:03.124801 6381 leader_election.cc:302] T 8f56a28665f342c5a77baf86dbe7d041 P 8315d22c56f7439a9e1fc82f153f018c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f9cb72be98941b5a8e504cbe34132b9, 8315d22c56f7439a9e1fc82f153f018c; no voters:
I20240501 07:59:03.125021 8193 raft_consensus.cc:2781] T 8f56a28665f342c5a77baf86dbe7d041 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:03.125090 8193 raft_consensus.cc:482] T 8f56a28665f342c5a77baf86dbe7d041 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:03.125144 8193 raft_consensus.cc:3037] T 8f56a28665f342c5a77baf86dbe7d041 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:03.125979 8193 raft_consensus.cc:504] T 8f56a28665f342c5a77baf86dbe7d041 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } }
I20240501 07:59:03.126287 8193 leader_election.cc:288] T 8f56a28665f342c5a77baf86dbe7d041 P 8315d22c56f7439a9e1fc82f153f018c [CANDIDATE]: Term 1 election: Requested vote from peers 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132:36773), 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129:46875)
I20240501 07:59:03.126583 6313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f56a28665f342c5a77baf86dbe7d041" candidate_uuid: "8315d22c56f7439a9e1fc82f153f018c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f"
I20240501 07:59:03.126691 6313 raft_consensus.cc:3037] T 8f56a28665f342c5a77baf86dbe7d041 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:03.126724 6708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f56a28665f342c5a77baf86dbe7d041" candidate_uuid: "8315d22c56f7439a9e1fc82f153f018c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f9cb72be98941b5a8e504cbe34132b9"
I20240501 07:59:03.126873 6708 raft_consensus.cc:3037] T 8f56a28665f342c5a77baf86dbe7d041 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:03.127499 6313 raft_consensus.cc:2454] T 8f56a28665f342c5a77baf86dbe7d041 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8315d22c56f7439a9e1fc82f153f018c in term 1.
I20240501 07:59:03.127606 6708 raft_consensus.cc:2454] T 8f56a28665f342c5a77baf86dbe7d041 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8315d22c56f7439a9e1fc82f153f018c in term 1.
I20240501 07:59:03.127766 6379 leader_election.cc:302] T 8f56a28665f342c5a77baf86dbe7d041 P 8315d22c56f7439a9e1fc82f153f018c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8315d22c56f7439a9e1fc82f153f018c, 8b8b3ad3aa8f4053b2167dd550f2086f; no voters:
I20240501 07:59:03.127971 8193 raft_consensus.cc:2781] T 8f56a28665f342c5a77baf86dbe7d041 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:03.128053 8193 raft_consensus.cc:686] T 8f56a28665f342c5a77baf86dbe7d041 P 8315d22c56f7439a9e1fc82f153f018c [term 1 LEADER]: Becoming Leader. State: Replica: 8315d22c56f7439a9e1fc82f153f018c, State: Running, Role: LEADER
I20240501 07:59:03.128192 8193 consensus_queue.cc:237] T 8f56a28665f342c5a77baf86dbe7d041 P 8315d22c56f7439a9e1fc82f153f018c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } }
I20240501 07:59:03.130049 6172 catalog_manager.cc:5484] T 8f56a28665f342c5a77baf86dbe7d041 P 8315d22c56f7439a9e1fc82f153f018c reported cstate change: term changed from 0 to 1, leader changed from <none> to 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130). New cstate: current_term: 1 leader_uuid: "8315d22c56f7439a9e1fc82f153f018c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } health_report { overall_health: UNKNOWN } } }
I20240501 07:59:03.207335 8223 raft_consensus.cc:482] T b914638d3a884ae4932316bdbd1fb4a4 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:03.207487 8223 raft_consensus.cc:504] T b914638d3a884ae4932316bdbd1fb4a4 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } }
I20240501 07:59:03.207923 8223 leader_election.cc:288] T b914638d3a884ae4932316bdbd1fb4a4 P 1f9cb72be98941b5a8e504cbe34132b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130:32941), 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129:46875)
I20240501 07:59:03.208302 6445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b914638d3a884ae4932316bdbd1fb4a4" candidate_uuid: "1f9cb72be98941b5a8e504cbe34132b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8315d22c56f7439a9e1fc82f153f018c" is_pre_election: true
I20240501 07:59:03.208403 6313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b914638d3a884ae4932316bdbd1fb4a4" candidate_uuid: "1f9cb72be98941b5a8e504cbe34132b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" is_pre_election: true
I20240501 07:59:03.208456 6445 raft_consensus.cc:2454] T b914638d3a884ae4932316bdbd1fb4a4 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f9cb72be98941b5a8e504cbe34132b9 in term 0.
I20240501 07:59:03.208555 6313 raft_consensus.cc:2454] T b914638d3a884ae4932316bdbd1fb4a4 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f9cb72be98941b5a8e504cbe34132b9 in term 0.
I20240501 07:59:03.208747 6643 leader_election.cc:302] T b914638d3a884ae4932316bdbd1fb4a4 P 1f9cb72be98941b5a8e504cbe34132b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f9cb72be98941b5a8e504cbe34132b9, 8315d22c56f7439a9e1fc82f153f018c; no voters:
I20240501 07:59:03.208981 8223 raft_consensus.cc:2781] T b914638d3a884ae4932316bdbd1fb4a4 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:03.209033 8223 raft_consensus.cc:482] T b914638d3a884ae4932316bdbd1fb4a4 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:03.209072 8223 raft_consensus.cc:3037] T b914638d3a884ae4932316bdbd1fb4a4 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:03.210115 8223 raft_consensus.cc:504] T b914638d3a884ae4932316bdbd1fb4a4 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } }
I20240501 07:59:03.210464 8223 leader_election.cc:288] T b914638d3a884ae4932316bdbd1fb4a4 P 1f9cb72be98941b5a8e504cbe34132b9 [CANDIDATE]: Term 1 election: Requested vote from peers 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130:32941), 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129:46875)
I20240501 07:59:03.210937 6313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b914638d3a884ae4932316bdbd1fb4a4" candidate_uuid: "1f9cb72be98941b5a8e504cbe34132b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f"
I20240501 07:59:03.210986 6445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b914638d3a884ae4932316bdbd1fb4a4" candidate_uuid: "1f9cb72be98941b5a8e504cbe34132b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8315d22c56f7439a9e1fc82f153f018c"
I20240501 07:59:03.211081 6313 raft_consensus.cc:3037] T b914638d3a884ae4932316bdbd1fb4a4 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:03.211089 6445 raft_consensus.cc:3037] T b914638d3a884ae4932316bdbd1fb4a4 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:03.212040 6445 raft_consensus.cc:2454] T b914638d3a884ae4932316bdbd1fb4a4 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f9cb72be98941b5a8e504cbe34132b9 in term 1.
I20240501 07:59:03.212040 6313 raft_consensus.cc:2454] T b914638d3a884ae4932316bdbd1fb4a4 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f9cb72be98941b5a8e504cbe34132b9 in term 1.
I20240501 07:59:03.212352 6641 leader_election.cc:302] T b914638d3a884ae4932316bdbd1fb4a4 P 1f9cb72be98941b5a8e504cbe34132b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f9cb72be98941b5a8e504cbe34132b9, 8b8b3ad3aa8f4053b2167dd550f2086f; no voters:
I20240501 07:59:03.212550 8223 raft_consensus.cc:2781] T b914638d3a884ae4932316bdbd1fb4a4 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:03.212641 8223 raft_consensus.cc:686] T b914638d3a884ae4932316bdbd1fb4a4 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 LEADER]: Becoming Leader. State: Replica: 1f9cb72be98941b5a8e504cbe34132b9, State: Running, Role: LEADER
I20240501 07:59:03.212775 8223 consensus_queue.cc:237] T b914638d3a884ae4932316bdbd1fb4a4 P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } }
I20240501 07:59:03.214860 6172 catalog_manager.cc:5484] T b914638d3a884ae4932316bdbd1fb4a4 P 1f9cb72be98941b5a8e504cbe34132b9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132). New cstate: current_term: 1 leader_uuid: "1f9cb72be98941b5a8e504cbe34132b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } health_report { overall_health: UNKNOWN } } }
W20240501 07:59:03.360339 8347 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:59:03.360597 8347 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:59:03.363734 8347 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_14'
I20240501 07:59:03.444268 6313 raft_consensus.cc:1261] T b914638d3a884ae4932316bdbd1fb4a4 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Refusing update from remote peer 1f9cb72be98941b5a8e504cbe34132b9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:03.445338 7959 consensus_queue.cc:1035] T b914638d3a884ae4932316bdbd1fb4a4 P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:03.445801 6708 raft_consensus.cc:1261] T 3c0ec93103d349b5b57853f4b4e9f8db P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Refusing update from remote peer 3b164cbbc00f4e8eb3254ddd349a3ff9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:03.446836 6708 raft_consensus.cc:1261] T 8f56a28665f342c5a77baf86dbe7d041 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Refusing update from remote peer 8315d22c56f7439a9e1fc82f153f018c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:03.446967 6313 raft_consensus.cc:1261] T 3c0ec93103d349b5b57853f4b4e9f8db P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Refusing update from remote peer 3b164cbbc00f4e8eb3254ddd349a3ff9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:03.447275 8253 consensus_queue.cc:1035] T 8f56a28665f342c5a77baf86dbe7d041 P 8315d22c56f7439a9e1fc82f153f018c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:03.447360 8235 consensus_queue.cc:1035] T 3c0ec93103d349b5b57853f4b4e9f8db P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:03.449227 8235 consensus_queue.cc:1035] T 3c0ec93103d349b5b57853f4b4e9f8db P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:03.452842 6315 raft_consensus.cc:1261] T 8f56a28665f342c5a77baf86dbe7d041 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Refusing update from remote peer 8315d22c56f7439a9e1fc82f153f018c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:03.453812 8193 consensus_queue.cc:1035] T 8f56a28665f342c5a77baf86dbe7d041 P 8315d22c56f7439a9e1fc82f153f018c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:03.482767 6446 raft_consensus.cc:1261] T b914638d3a884ae4932316bdbd1fb4a4 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Refusing update from remote peer 1f9cb72be98941b5a8e504cbe34132b9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:03.483173 7959 consensus_queue.cc:1035] T b914638d3a884ae4932316bdbd1fb4a4 P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 122.318 ms
time per row: 0.0611588 ms
I20240501 07:59:03.529177 6170 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55950:
name: "rebalance_test_table_15"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 07:59:03.535130 6556 tablet_service.cc:1467] Processing CreateTablet for tablet 2e540f389b2a4aef93a020f549d11b67 (DEFAULT_TABLE table=rebalance_test_table_15 [id=ecf5cb7235314f20bbb522737ba7fd3c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:03.535130 6425 tablet_service.cc:1467] Processing CreateTablet for tablet 2e540f389b2a4aef93a020f549d11b67 (DEFAULT_TABLE table=rebalance_test_table_15 [id=ecf5cb7235314f20bbb522737ba7fd3c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:03.535272 6555 tablet_service.cc:1467] Processing CreateTablet for tablet 3b9b71d44f754353b2321e8c36dd4dfd (DEFAULT_TABLE table=rebalance_test_table_15 [id=ecf5cb7235314f20bbb522737ba7fd3c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:03.535486 6425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e540f389b2a4aef93a020f549d11b67. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:03.535491 6556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e540f389b2a4aef93a020f549d11b67. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:03.535895 6555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b9b71d44f754353b2321e8c36dd4dfd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:03.537178 6687 tablet_service.cc:1467] Processing CreateTablet for tablet 3b9b71d44f754353b2321e8c36dd4dfd (DEFAULT_TABLE table=rebalance_test_table_15 [id=ecf5cb7235314f20bbb522737ba7fd3c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:03.537456 6687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b9b71d44f754353b2321e8c36dd4dfd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:03.537694 8334 tablet_bootstrap.cc:492] T 2e540f389b2a4aef93a020f549d11b67 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Bootstrap starting.
I20240501 07:59:03.537917 8331 tablet_bootstrap.cc:492] T 2e540f389b2a4aef93a020f549d11b67 P 8315d22c56f7439a9e1fc82f153f018c: Bootstrap starting.
I20240501 07:59:03.537911 6686 tablet_service.cc:1467] Processing CreateTablet for tablet 271e3c5cf9934eaead7fa1ba942084be (DEFAULT_TABLE table=rebalance_test_table_15 [id=ecf5cb7235314f20bbb522737ba7fd3c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:03.538141 6686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 271e3c5cf9934eaead7fa1ba942084be. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:03.538507 8334 tablet_bootstrap.cc:654] T 2e540f389b2a4aef93a020f549d11b67 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:03.538797 8331 tablet_bootstrap.cc:654] T 2e540f389b2a4aef93a020f549d11b67 P 8315d22c56f7439a9e1fc82f153f018c: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:03.539543 8332 tablet_bootstrap.cc:492] T 3b9b71d44f754353b2321e8c36dd4dfd P 1f9cb72be98941b5a8e504cbe34132b9: Bootstrap starting.
I20240501 07:59:03.539894 6425 tablet_service.cc:1467] Processing CreateTablet for tablet 271e3c5cf9934eaead7fa1ba942084be (DEFAULT_TABLE table=rebalance_test_table_15 [id=ecf5cb7235314f20bbb522737ba7fd3c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:03.540139 6425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 271e3c5cf9934eaead7fa1ba942084be. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:03.540215 6293 tablet_service.cc:1467] Processing CreateTablet for tablet 3b9b71d44f754353b2321e8c36dd4dfd (DEFAULT_TABLE table=rebalance_test_table_15 [id=ecf5cb7235314f20bbb522737ba7fd3c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:03.540316 8332 tablet_bootstrap.cc:654] T 3b9b71d44f754353b2321e8c36dd4dfd P 1f9cb72be98941b5a8e504cbe34132b9: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:03.540467 6293 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b9b71d44f754353b2321e8c36dd4dfd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:03.540910 6295 tablet_service.cc:1467] Processing CreateTablet for tablet 2e540f389b2a4aef93a020f549d11b67 (DEFAULT_TABLE table=rebalance_test_table_15 [id=ecf5cb7235314f20bbb522737ba7fd3c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:03.541129 6295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e540f389b2a4aef93a020f549d11b67. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:03.542640 8335 tablet_bootstrap.cc:492] T 3b9b71d44f754353b2321e8c36dd4dfd P 8b8b3ad3aa8f4053b2167dd550f2086f: Bootstrap starting.
I20240501 07:59:03.542904 8334 tablet_bootstrap.cc:492] T 2e540f389b2a4aef93a020f549d11b67 P 3b164cbbc00f4e8eb3254ddd349a3ff9: No bootstrap required, opened a new log
I20240501 07:59:03.543000 8334 ts_tablet_manager.cc:1367] T 2e540f389b2a4aef93a020f549d11b67 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240501 07:59:03.543449 8335 tablet_bootstrap.cc:654] T 3b9b71d44f754353b2321e8c36dd4dfd P 8b8b3ad3aa8f4053b2167dd550f2086f: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:03.543448 8334 raft_consensus.cc:348] T 2e540f389b2a4aef93a020f549d11b67 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } }
I20240501 07:59:03.543570 8334 raft_consensus.cc:374] T 2e540f389b2a4aef93a020f549d11b67 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:03.543618 8334 raft_consensus.cc:729] T 2e540f389b2a4aef93a020f549d11b67 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b164cbbc00f4e8eb3254ddd349a3ff9, State: Initialized, Role: FOLLOWER
I20240501 07:59:03.543758 8334 consensus_queue.cc:260] T 2e540f389b2a4aef93a020f549d11b67 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } }
I20240501 07:59:03.543915 8332 tablet_bootstrap.cc:492] T 3b9b71d44f754353b2321e8c36dd4dfd P 1f9cb72be98941b5a8e504cbe34132b9: No bootstrap required, opened a new log
I20240501 07:59:03.544001 8332 ts_tablet_manager.cc:1367] T 3b9b71d44f754353b2321e8c36dd4dfd P 1f9cb72be98941b5a8e504cbe34132b9: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20240501 07:59:03.544092 8334 ts_tablet_manager.cc:1398] T 2e540f389b2a4aef93a020f549d11b67 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:59:03.544260 8334 tablet_bootstrap.cc:492] T 3b9b71d44f754353b2321e8c36dd4dfd P 3b164cbbc00f4e8eb3254ddd349a3ff9: Bootstrap starting.
I20240501 07:59:03.544504 8335 tablet_bootstrap.cc:492] T 3b9b71d44f754353b2321e8c36dd4dfd P 8b8b3ad3aa8f4053b2167dd550f2086f: No bootstrap required, opened a new log
I20240501 07:59:03.544489 8332 raft_consensus.cc:348] T 3b9b71d44f754353b2321e8c36dd4dfd P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } }
I20240501 07:59:03.544572 8335 ts_tablet_manager.cc:1367] T 3b9b71d44f754353b2321e8c36dd4dfd P 8b8b3ad3aa8f4053b2167dd550f2086f: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240501 07:59:03.544605 8332 raft_consensus.cc:374] T 3b9b71d44f754353b2321e8c36dd4dfd P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:03.544658 8332 raft_consensus.cc:729] T 3b9b71d44f754353b2321e8c36dd4dfd P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f9cb72be98941b5a8e504cbe34132b9, State: Initialized, Role: FOLLOWER
I20240501 07:59:03.544783 8332 consensus_queue.cc:260] T 3b9b71d44f754353b2321e8c36dd4dfd P 1f9cb72be98941b5a8e504cbe34132b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } }
I20240501 07:59:03.544896 8334 tablet_bootstrap.cc:654] T 3b9b71d44f754353b2321e8c36dd4dfd P 3b164cbbc00f4e8eb3254ddd349a3ff9: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:03.545028 8335 raft_consensus.cc:348] T 3b9b71d44f754353b2321e8c36dd4dfd P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } }
I20240501 07:59:03.545122 8335 raft_consensus.cc:374] T 3b9b71d44f754353b2321e8c36dd4dfd P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:03.545125 8332 ts_tablet_manager.cc:1398] T 3b9b71d44f754353b2321e8c36dd4dfd P 1f9cb72be98941b5a8e504cbe34132b9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:03.545156 8335 raft_consensus.cc:729] T 3b9b71d44f754353b2321e8c36dd4dfd P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b8b3ad3aa8f4053b2167dd550f2086f, State: Initialized, Role: FOLLOWER
I20240501 07:59:03.545296 8332 tablet_bootstrap.cc:492] T 271e3c5cf9934eaead7fa1ba942084be P 1f9cb72be98941b5a8e504cbe34132b9: Bootstrap starting.
I20240501 07:59:03.545272 8335 consensus_queue.cc:260] T 3b9b71d44f754353b2321e8c36dd4dfd P 8b8b3ad3aa8f4053b2167dd550f2086f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } }
I20240501 07:59:03.545531 8335 ts_tablet_manager.cc:1398] T 3b9b71d44f754353b2321e8c36dd4dfd P 8b8b3ad3aa8f4053b2167dd550f2086f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:03.545698 8335 tablet_bootstrap.cc:492] T 2e540f389b2a4aef93a020f549d11b67 P 8b8b3ad3aa8f4053b2167dd550f2086f: Bootstrap starting.
I20240501 07:59:03.546936 6555 tablet_service.cc:1467] Processing CreateTablet for tablet 271e3c5cf9934eaead7fa1ba942084be (DEFAULT_TABLE table=rebalance_test_table_15 [id=ecf5cb7235314f20bbb522737ba7fd3c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:03.547180 6555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 271e3c5cf9934eaead7fa1ba942084be. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:03.547729 8335 tablet_bootstrap.cc:654] T 2e540f389b2a4aef93a020f549d11b67 P 8b8b3ad3aa8f4053b2167dd550f2086f: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:03.549214 8332 tablet_bootstrap.cc:654] T 271e3c5cf9934eaead7fa1ba942084be P 1f9cb72be98941b5a8e504cbe34132b9: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:03.550736 8331 tablet_bootstrap.cc:492] T 2e540f389b2a4aef93a020f549d11b67 P 8315d22c56f7439a9e1fc82f153f018c: No bootstrap required, opened a new log
I20240501 07:59:03.550868 8331 ts_tablet_manager.cc:1367] T 2e540f389b2a4aef93a020f549d11b67 P 8315d22c56f7439a9e1fc82f153f018c: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20240501 07:59:03.551385 8332 tablet_bootstrap.cc:492] T 271e3c5cf9934eaead7fa1ba942084be P 1f9cb72be98941b5a8e504cbe34132b9: No bootstrap required, opened a new log
I20240501 07:59:03.551468 8332 ts_tablet_manager.cc:1367] T 271e3c5cf9934eaead7fa1ba942084be P 1f9cb72be98941b5a8e504cbe34132b9: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 07:59:03.551508 8331 raft_consensus.cc:348] T 2e540f389b2a4aef93a020f549d11b67 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } }
I20240501 07:59:03.551630 8331 raft_consensus.cc:374] T 2e540f389b2a4aef93a020f549d11b67 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:03.551669 8331 raft_consensus.cc:729] T 2e540f389b2a4aef93a020f549d11b67 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8315d22c56f7439a9e1fc82f153f018c, State: Initialized, Role: FOLLOWER
I20240501 07:59:03.551797 8331 consensus_queue.cc:260] T 2e540f389b2a4aef93a020f549d11b67 P 8315d22c56f7439a9e1fc82f153f018c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } }
I20240501 07:59:03.551908 8332 raft_consensus.cc:348] T 271e3c5cf9934eaead7fa1ba942084be P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:59:03.552000 8332 raft_consensus.cc:374] T 271e3c5cf9934eaead7fa1ba942084be P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:03.552044 8332 raft_consensus.cc:729] T 271e3c5cf9934eaead7fa1ba942084be P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f9cb72be98941b5a8e504cbe34132b9, State: Initialized, Role: FOLLOWER
I20240501 07:59:03.552119 8331 ts_tablet_manager.cc:1398] T 2e540f389b2a4aef93a020f549d11b67 P 8315d22c56f7439a9e1fc82f153f018c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:59:03.552166 8332 consensus_queue.cc:260] T 271e3c5cf9934eaead7fa1ba942084be P 1f9cb72be98941b5a8e504cbe34132b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:59:03.552297 8331 tablet_bootstrap.cc:492] T 271e3c5cf9934eaead7fa1ba942084be P 8315d22c56f7439a9e1fc82f153f018c: Bootstrap starting.
I20240501 07:59:03.552433 8332 ts_tablet_manager.cc:1398] T 271e3c5cf9934eaead7fa1ba942084be P 1f9cb72be98941b5a8e504cbe34132b9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:03.553400 8331 tablet_bootstrap.cc:654] T 271e3c5cf9934eaead7fa1ba942084be P 8315d22c56f7439a9e1fc82f153f018c: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:03.554555 8334 tablet_bootstrap.cc:492] T 3b9b71d44f754353b2321e8c36dd4dfd P 3b164cbbc00f4e8eb3254ddd349a3ff9: No bootstrap required, opened a new log
I20240501 07:59:03.554648 8334 ts_tablet_manager.cc:1367] T 3b9b71d44f754353b2321e8c36dd4dfd P 3b164cbbc00f4e8eb3254ddd349a3ff9: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240501 07:59:03.554841 8335 tablet_bootstrap.cc:492] T 2e540f389b2a4aef93a020f549d11b67 P 8b8b3ad3aa8f4053b2167dd550f2086f: No bootstrap required, opened a new log
I20240501 07:59:03.554929 8335 ts_tablet_manager.cc:1367] T 2e540f389b2a4aef93a020f549d11b67 P 8b8b3ad3aa8f4053b2167dd550f2086f: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.001s
I20240501 07:59:03.555111 8334 raft_consensus.cc:348] T 3b9b71d44f754353b2321e8c36dd4dfd P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } }
I20240501 07:59:03.555209 8334 raft_consensus.cc:374] T 3b9b71d44f754353b2321e8c36dd4dfd P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:03.555241 8334 raft_consensus.cc:729] T 3b9b71d44f754353b2321e8c36dd4dfd P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b164cbbc00f4e8eb3254ddd349a3ff9, State: Initialized, Role: FOLLOWER
I20240501 07:59:03.555375 8334 consensus_queue.cc:260] T 3b9b71d44f754353b2321e8c36dd4dfd P 3b164cbbc00f4e8eb3254ddd349a3ff9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } }
I20240501 07:59:03.555437 8335 raft_consensus.cc:348] T 2e540f389b2a4aef93a020f549d11b67 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } }
I20240501 07:59:03.555541 8335 raft_consensus.cc:374] T 2e540f389b2a4aef93a020f549d11b67 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:03.555572 8335 raft_consensus.cc:729] T 2e540f389b2a4aef93a020f549d11b67 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b8b3ad3aa8f4053b2167dd550f2086f, State: Initialized, Role: FOLLOWER
I20240501 07:59:03.555675 8334 ts_tablet_manager.cc:1398] T 3b9b71d44f754353b2321e8c36dd4dfd P 3b164cbbc00f4e8eb3254ddd349a3ff9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:03.555689 8335 consensus_queue.cc:260] T 2e540f389b2a4aef93a020f549d11b67 P 8b8b3ad3aa8f4053b2167dd550f2086f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } }
I20240501 07:59:03.555838 8334 tablet_bootstrap.cc:492] T 271e3c5cf9934eaead7fa1ba942084be P 3b164cbbc00f4e8eb3254ddd349a3ff9: Bootstrap starting.
I20240501 07:59:03.555968 8335 ts_tablet_manager.cc:1398] T 2e540f389b2a4aef93a020f549d11b67 P 8b8b3ad3aa8f4053b2167dd550f2086f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:59:03.556743 8334 tablet_bootstrap.cc:654] T 271e3c5cf9934eaead7fa1ba942084be P 3b164cbbc00f4e8eb3254ddd349a3ff9: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:03.557996 8331 tablet_bootstrap.cc:492] T 271e3c5cf9934eaead7fa1ba942084be P 8315d22c56f7439a9e1fc82f153f018c: No bootstrap required, opened a new log
I20240501 07:59:03.558104 8331 ts_tablet_manager.cc:1367] T 271e3c5cf9934eaead7fa1ba942084be P 8315d22c56f7439a9e1fc82f153f018c: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 07:59:03.558574 8331 raft_consensus.cc:348] T 271e3c5cf9934eaead7fa1ba942084be P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:59:03.558688 8331 raft_consensus.cc:374] T 271e3c5cf9934eaead7fa1ba942084be P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:03.558722 8331 raft_consensus.cc:729] T 271e3c5cf9934eaead7fa1ba942084be P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8315d22c56f7439a9e1fc82f153f018c, State: Initialized, Role: FOLLOWER
I20240501 07:59:03.558843 8331 consensus_queue.cc:260] T 271e3c5cf9934eaead7fa1ba942084be P 8315d22c56f7439a9e1fc82f153f018c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:59:03.559140 8331 ts_tablet_manager.cc:1398] T 271e3c5cf9934eaead7fa1ba942084be P 8315d22c56f7439a9e1fc82f153f018c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:03.560485 8334 tablet_bootstrap.cc:492] T 271e3c5cf9934eaead7fa1ba942084be P 3b164cbbc00f4e8eb3254ddd349a3ff9: No bootstrap required, opened a new log
I20240501 07:59:03.560562 8334 ts_tablet_manager.cc:1367] T 271e3c5cf9934eaead7fa1ba942084be P 3b164cbbc00f4e8eb3254ddd349a3ff9: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240501 07:59:03.560927 8334 raft_consensus.cc:348] T 271e3c5cf9934eaead7fa1ba942084be P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:59:03.561071 8334 raft_consensus.cc:374] T 271e3c5cf9934eaead7fa1ba942084be P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:03.561106 8334 raft_consensus.cc:729] T 271e3c5cf9934eaead7fa1ba942084be P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b164cbbc00f4e8eb3254ddd349a3ff9, State: Initialized, Role: FOLLOWER
I20240501 07:59:03.561244 8334 consensus_queue.cc:260] T 271e3c5cf9934eaead7fa1ba942084be P 3b164cbbc00f4e8eb3254ddd349a3ff9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:59:03.561506 8334 ts_tablet_manager.cc:1398] T 271e3c5cf9934eaead7fa1ba942084be P 3b164cbbc00f4e8eb3254ddd349a3ff9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:03.590698 8193 raft_consensus.cc:482] T 271e3c5cf9934eaead7fa1ba942084be P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:03.590828 8193 raft_consensus.cc:504] T 271e3c5cf9934eaead7fa1ba942084be P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:59:03.591241 8193 leader_election.cc:288] T 271e3c5cf9934eaead7fa1ba942084be P 8315d22c56f7439a9e1fc82f153f018c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131:34471), 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132:36773)
I20240501 07:59:03.591732 6708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "271e3c5cf9934eaead7fa1ba942084be" candidate_uuid: "8315d22c56f7439a9e1fc82f153f018c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f9cb72be98941b5a8e504cbe34132b9" is_pre_election: true
I20240501 07:59:03.591732 6575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "271e3c5cf9934eaead7fa1ba942084be" candidate_uuid: "8315d22c56f7439a9e1fc82f153f018c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" is_pre_election: true
I20240501 07:59:03.591892 6708 raft_consensus.cc:2454] T 271e3c5cf9934eaead7fa1ba942084be P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8315d22c56f7439a9e1fc82f153f018c in term 0.
I20240501 07:59:03.591926 6575 raft_consensus.cc:2454] T 271e3c5cf9934eaead7fa1ba942084be P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8315d22c56f7439a9e1fc82f153f018c in term 0.
I20240501 07:59:03.592216 6381 leader_election.cc:302] T 271e3c5cf9934eaead7fa1ba942084be P 8315d22c56f7439a9e1fc82f153f018c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f9cb72be98941b5a8e504cbe34132b9, 8315d22c56f7439a9e1fc82f153f018c; no voters:
I20240501 07:59:03.592372 8193 raft_consensus.cc:2781] T 271e3c5cf9934eaead7fa1ba942084be P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:03.592438 8193 raft_consensus.cc:482] T 271e3c5cf9934eaead7fa1ba942084be P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:03.592470 8193 raft_consensus.cc:3037] T 271e3c5cf9934eaead7fa1ba942084be P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:03.593395 8193 raft_consensus.cc:504] T 271e3c5cf9934eaead7fa1ba942084be P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:59:03.593750 8193 leader_election.cc:288] T 271e3c5cf9934eaead7fa1ba942084be P 8315d22c56f7439a9e1fc82f153f018c [CANDIDATE]: Term 1 election: Requested vote from peers 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131:34471), 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132:36773)
I20240501 07:59:03.594094 6575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "271e3c5cf9934eaead7fa1ba942084be" candidate_uuid: "8315d22c56f7439a9e1fc82f153f018c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9"
I20240501 07:59:03.594094 6707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "271e3c5cf9934eaead7fa1ba942084be" candidate_uuid: "8315d22c56f7439a9e1fc82f153f018c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f9cb72be98941b5a8e504cbe34132b9"
I20240501 07:59:03.594209 6707 raft_consensus.cc:3037] T 271e3c5cf9934eaead7fa1ba942084be P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:03.594209 6575 raft_consensus.cc:3037] T 271e3c5cf9934eaead7fa1ba942084be P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:03.595168 6575 raft_consensus.cc:2454] T 271e3c5cf9934eaead7fa1ba942084be P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8315d22c56f7439a9e1fc82f153f018c in term 1.
I20240501 07:59:03.595170 6707 raft_consensus.cc:2454] T 271e3c5cf9934eaead7fa1ba942084be P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8315d22c56f7439a9e1fc82f153f018c in term 1.
I20240501 07:59:03.595467 6381 leader_election.cc:302] T 271e3c5cf9934eaead7fa1ba942084be P 8315d22c56f7439a9e1fc82f153f018c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f9cb72be98941b5a8e504cbe34132b9, 8315d22c56f7439a9e1fc82f153f018c; no voters:
I20240501 07:59:03.595686 8193 raft_consensus.cc:2781] T 271e3c5cf9934eaead7fa1ba942084be P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:03.595778 8193 raft_consensus.cc:686] T 271e3c5cf9934eaead7fa1ba942084be P 8315d22c56f7439a9e1fc82f153f018c [term 1 LEADER]: Becoming Leader. State: Replica: 8315d22c56f7439a9e1fc82f153f018c, State: Running, Role: LEADER
I20240501 07:59:03.595927 8193 consensus_queue.cc:237] T 271e3c5cf9934eaead7fa1ba942084be P 8315d22c56f7439a9e1fc82f153f018c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:59:03.598026 6170 catalog_manager.cc:5484] T 271e3c5cf9934eaead7fa1ba942084be P 8315d22c56f7439a9e1fc82f153f018c reported cstate change: term changed from 0 to 1, leader changed from <none> to 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130). New cstate: current_term: 1 leader_uuid: "8315d22c56f7439a9e1fc82f153f018c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } health_report { overall_health: UNKNOWN } } }
I20240501 07:59:03.637008 8385 raft_consensus.cc:482] T 3b9b71d44f754353b2321e8c36dd4dfd P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:03.637176 8385 raft_consensus.cc:504] T 3b9b71d44f754353b2321e8c36dd4dfd P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } }
I20240501 07:59:03.637653 8385 leader_election.cc:288] T 3b9b71d44f754353b2321e8c36dd4dfd P 1f9cb72be98941b5a8e504cbe34132b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131:34471), 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129:46875)
I20240501 07:59:03.638067 6575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b9b71d44f754353b2321e8c36dd4dfd" candidate_uuid: "1f9cb72be98941b5a8e504cbe34132b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" is_pre_election: true
I20240501 07:59:03.638131 6313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b9b71d44f754353b2321e8c36dd4dfd" candidate_uuid: "1f9cb72be98941b5a8e504cbe34132b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" is_pre_election: true
I20240501 07:59:03.638193 6575 raft_consensus.cc:2454] T 3b9b71d44f754353b2321e8c36dd4dfd P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f9cb72be98941b5a8e504cbe34132b9 in term 0.
I20240501 07:59:03.638273 6313 raft_consensus.cc:2454] T 3b9b71d44f754353b2321e8c36dd4dfd P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f9cb72be98941b5a8e504cbe34132b9 in term 0.
I20240501 07:59:03.638482 6644 leader_election.cc:302] T 3b9b71d44f754353b2321e8c36dd4dfd P 1f9cb72be98941b5a8e504cbe34132b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f9cb72be98941b5a8e504cbe34132b9, 3b164cbbc00f4e8eb3254ddd349a3ff9; no voters:
I20240501 07:59:03.638664 8385 raft_consensus.cc:2781] T 3b9b71d44f754353b2321e8c36dd4dfd P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:03.638724 8385 raft_consensus.cc:482] T 3b9b71d44f754353b2321e8c36dd4dfd P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:03.638758 8385 raft_consensus.cc:3037] T 3b9b71d44f754353b2321e8c36dd4dfd P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:03.639597 8385 raft_consensus.cc:504] T 3b9b71d44f754353b2321e8c36dd4dfd P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } }
I20240501 07:59:03.639927 8385 leader_election.cc:288] T 3b9b71d44f754353b2321e8c36dd4dfd P 1f9cb72be98941b5a8e504cbe34132b9 [CANDIDATE]: Term 1 election: Requested vote from peers 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131:34471), 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129:46875)
I20240501 07:59:03.640193 6575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b9b71d44f754353b2321e8c36dd4dfd" candidate_uuid: "1f9cb72be98941b5a8e504cbe34132b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9"
I20240501 07:59:03.640233 6313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b9b71d44f754353b2321e8c36dd4dfd" candidate_uuid: "1f9cb72be98941b5a8e504cbe34132b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f"
I20240501 07:59:03.640291 6575 raft_consensus.cc:3037] T 3b9b71d44f754353b2321e8c36dd4dfd P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:03.640347 6313 raft_consensus.cc:3037] T 3b9b71d44f754353b2321e8c36dd4dfd P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:03.641110 6313 raft_consensus.cc:2454] T 3b9b71d44f754353b2321e8c36dd4dfd P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f9cb72be98941b5a8e504cbe34132b9 in term 1.
I20240501 07:59:03.641299 6575 raft_consensus.cc:2454] T 3b9b71d44f754353b2321e8c36dd4dfd P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f9cb72be98941b5a8e504cbe34132b9 in term 1.
I20240501 07:59:03.641402 6641 leader_election.cc:302] T 3b9b71d44f754353b2321e8c36dd4dfd P 1f9cb72be98941b5a8e504cbe34132b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f9cb72be98941b5a8e504cbe34132b9, 8b8b3ad3aa8f4053b2167dd550f2086f; no voters:
I20240501 07:59:03.641577 8385 raft_consensus.cc:2781] T 3b9b71d44f754353b2321e8c36dd4dfd P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:03.641656 8385 raft_consensus.cc:686] T 3b9b71d44f754353b2321e8c36dd4dfd P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 LEADER]: Becoming Leader. State: Replica: 1f9cb72be98941b5a8e504cbe34132b9, State: Running, Role: LEADER
I20240501 07:59:03.641800 8385 consensus_queue.cc:237] T 3b9b71d44f754353b2321e8c36dd4dfd P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } }
I20240501 07:59:03.643777 6170 catalog_manager.cc:5484] T 3b9b71d44f754353b2321e8c36dd4dfd P 1f9cb72be98941b5a8e504cbe34132b9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132). New cstate: current_term: 1 leader_uuid: "1f9cb72be98941b5a8e504cbe34132b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } health_report { overall_health: UNKNOWN } } }
I20240501 07:59:03.779048 8105 raft_consensus.cc:482] T 2e540f389b2a4aef93a020f549d11b67 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:03.779206 8105 raft_consensus.cc:504] T 2e540f389b2a4aef93a020f549d11b67 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } }
I20240501 07:59:03.779640 8105 leader_election.cc:288] T 2e540f389b2a4aef93a020f549d11b67 P 8b8b3ad3aa8f4053b2167dd550f2086f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130:32941), 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131:34471)
I20240501 07:59:03.780102 6445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e540f389b2a4aef93a020f549d11b67" candidate_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8315d22c56f7439a9e1fc82f153f018c" is_pre_election: true
I20240501 07:59:03.780104 6575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e540f389b2a4aef93a020f549d11b67" candidate_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" is_pre_election: true
I20240501 07:59:03.780257 6575 raft_consensus.cc:2454] T 2e540f389b2a4aef93a020f549d11b67 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b8b3ad3aa8f4053b2167dd550f2086f in term 0.
I20240501 07:59:03.780267 6445 raft_consensus.cc:2454] T 2e540f389b2a4aef93a020f549d11b67 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b8b3ad3aa8f4053b2167dd550f2086f in term 0.
I20240501 07:59:03.780556 6251 leader_election.cc:302] T 2e540f389b2a4aef93a020f549d11b67 P 8b8b3ad3aa8f4053b2167dd550f2086f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b164cbbc00f4e8eb3254ddd349a3ff9, 8b8b3ad3aa8f4053b2167dd550f2086f; no voters:
I20240501 07:59:03.780741 8105 raft_consensus.cc:2781] T 2e540f389b2a4aef93a020f549d11b67 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:03.780805 8105 raft_consensus.cc:482] T 2e540f389b2a4aef93a020f549d11b67 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:03.780843 8105 raft_consensus.cc:3037] T 2e540f389b2a4aef93a020f549d11b67 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:03.781879 8105 raft_consensus.cc:504] T 2e540f389b2a4aef93a020f549d11b67 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } }
I20240501 07:59:03.782280 8105 leader_election.cc:288] T 2e540f389b2a4aef93a020f549d11b67 P 8b8b3ad3aa8f4053b2167dd550f2086f [CANDIDATE]: Term 1 election: Requested vote from peers 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130:32941), 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131:34471)
I20240501 07:59:03.782577 6445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e540f389b2a4aef93a020f549d11b67" candidate_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8315d22c56f7439a9e1fc82f153f018c"
I20240501 07:59:03.782698 6445 raft_consensus.cc:3037] T 2e540f389b2a4aef93a020f549d11b67 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:03.783483 6445 raft_consensus.cc:2454] T 2e540f389b2a4aef93a020f549d11b67 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b8b3ad3aa8f4053b2167dd550f2086f in term 1.
I20240501 07:59:03.783720 6575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e540f389b2a4aef93a020f549d11b67" candidate_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9"
I20240501 07:59:03.783761 6250 leader_election.cc:302] T 2e540f389b2a4aef93a020f549d11b67 P 8b8b3ad3aa8f4053b2167dd550f2086f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8315d22c56f7439a9e1fc82f153f018c, 8b8b3ad3aa8f4053b2167dd550f2086f; no voters:
I20240501 07:59:03.783860 6575 raft_consensus.cc:3037] T 2e540f389b2a4aef93a020f549d11b67 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:03.783924 8105 raft_consensus.cc:2781] T 2e540f389b2a4aef93a020f549d11b67 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:03.784009 8105 raft_consensus.cc:686] T 2e540f389b2a4aef93a020f549d11b67 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 LEADER]: Becoming Leader. State: Replica: 8b8b3ad3aa8f4053b2167dd550f2086f, State: Running, Role: LEADER
I20240501 07:59:03.784173 8105 consensus_queue.cc:237] T 2e540f389b2a4aef93a020f549d11b67 P 8b8b3ad3aa8f4053b2167dd550f2086f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } }
I20240501 07:59:03.784850 6575 raft_consensus.cc:2454] T 2e540f389b2a4aef93a020f549d11b67 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b8b3ad3aa8f4053b2167dd550f2086f in term 1.
I20240501 07:59:03.786252 6170 catalog_manager.cc:5484] T 2e540f389b2a4aef93a020f549d11b67 P 8b8b3ad3aa8f4053b2167dd550f2086f reported cstate change: term changed from 0 to 1, leader changed from <none> to 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129). New cstate: current_term: 1 leader_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } health_report { overall_health: UNKNOWN } } }
W20240501 07:59:03.955590 8400 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:59:03.955863 8400 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:59:03.959105 8400 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_15'
I20240501 07:59:03.996354 8193 consensus_queue.cc:1035] T 271e3c5cf9934eaead7fa1ba942084be P 8315d22c56f7439a9e1fc82f153f018c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:04.011240 8193 consensus_queue.cc:1035] T 271e3c5cf9934eaead7fa1ba942084be P 8315d22c56f7439a9e1fc82f153f018c [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:04.028622 8385 consensus_queue.cc:1035] T 3b9b71d44f754353b2321e8c36dd4dfd P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:04.040903 7959 consensus_queue.cc:1035] T 3b9b71d44f754353b2321e8c36dd4dfd P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:04.091935 6577 raft_consensus.cc:1261] T 2e540f389b2a4aef93a020f549d11b67 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Refusing update from remote peer 8b8b3ad3aa8f4053b2167dd550f2086f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:04.092415 8383 consensus_queue.cc:1035] T 2e540f389b2a4aef93a020f549d11b67 P 8b8b3ad3aa8f4053b2167dd550f2086f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:04.092700 6445 raft_consensus.cc:1261] T 2e540f389b2a4aef93a020f549d11b67 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Refusing update from remote peer 8b8b3ad3aa8f4053b2167dd550f2086f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:04.093245 8383 consensus_queue.cc:1035] T 2e540f389b2a4aef93a020f549d11b67 P 8b8b3ad3aa8f4053b2167dd550f2086f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 150.401 ms
time per row: 0.0752005 ms
I20240501 07:59:04.153743 6167 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55950:
name: "rebalance_test_table_16"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 07:59:04.159322 6555 tablet_service.cc:1467] Processing CreateTablet for tablet 33b3834db0cd4407ae7c26e044459e87 (DEFAULT_TABLE table=rebalance_test_table_16 [id=bdae856e60144776aae709dddb837795]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:04.159490 6295 tablet_service.cc:1467] Processing CreateTablet for tablet 33b3834db0cd4407ae7c26e044459e87 (DEFAULT_TABLE table=rebalance_test_table_16 [id=bdae856e60144776aae709dddb837795]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:04.159657 6555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33b3834db0cd4407ae7c26e044459e87. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:04.159797 6295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33b3834db0cd4407ae7c26e044459e87. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:04.160176 6425 tablet_service.cc:1467] Processing CreateTablet for tablet 33b3834db0cd4407ae7c26e044459e87 (DEFAULT_TABLE table=rebalance_test_table_16 [id=bdae856e60144776aae709dddb837795]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:04.160234 6426 tablet_service.cc:1467] Processing CreateTablet for tablet af8baea378be41f1806ea2cae61764bb (DEFAULT_TABLE table=rebalance_test_table_16 [id=bdae856e60144776aae709dddb837795]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:04.160420 6425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33b3834db0cd4407ae7c26e044459e87. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:04.160843 6426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af8baea378be41f1806ea2cae61764bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:04.161348 6294 tablet_service.cc:1467] Processing CreateTablet for tablet 3b04063d87be40f7ac1b46715d457983 (DEFAULT_TABLE table=rebalance_test_table_16 [id=bdae856e60144776aae709dddb837795]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:04.161564 6294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b04063d87be40f7ac1b46715d457983. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:04.162324 6293 tablet_service.cc:1467] Processing CreateTablet for tablet af8baea378be41f1806ea2cae61764bb (DEFAULT_TABLE table=rebalance_test_table_16 [id=bdae856e60144776aae709dddb837795]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:04.162551 6293 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af8baea378be41f1806ea2cae61764bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:04.163064 8454 tablet_bootstrap.cc:492] T 33b3834db0cd4407ae7c26e044459e87 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Bootstrap starting.
I20240501 07:59:04.164036 6686 tablet_service.cc:1467] Processing CreateTablet for tablet af8baea378be41f1806ea2cae61764bb (DEFAULT_TABLE table=rebalance_test_table_16 [id=bdae856e60144776aae709dddb837795]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:04.164302 6686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af8baea378be41f1806ea2cae61764bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:04.164316 8454 tablet_bootstrap.cc:654] T 33b3834db0cd4407ae7c26e044459e87 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:04.164690 6687 tablet_service.cc:1467] Processing CreateTablet for tablet 3b04063d87be40f7ac1b46715d457983 (DEFAULT_TABLE table=rebalance_test_table_16 [id=bdae856e60144776aae709dddb837795]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:04.164939 6687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b04063d87be40f7ac1b46715d457983. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:04.165832 8456 tablet_bootstrap.cc:492] T 33b3834db0cd4407ae7c26e044459e87 P 8315d22c56f7439a9e1fc82f153f018c: Bootstrap starting.
I20240501 07:59:04.166836 8458 tablet_bootstrap.cc:492] T af8baea378be41f1806ea2cae61764bb P 1f9cb72be98941b5a8e504cbe34132b9: Bootstrap starting.
I20240501 07:59:04.166889 8456 tablet_bootstrap.cc:654] T 33b3834db0cd4407ae7c26e044459e87 P 8315d22c56f7439a9e1fc82f153f018c: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:04.167586 8455 tablet_bootstrap.cc:492] T 33b3834db0cd4407ae7c26e044459e87 P 8b8b3ad3aa8f4053b2167dd550f2086f: Bootstrap starting.
I20240501 07:59:04.167538 6555 tablet_service.cc:1467] Processing CreateTablet for tablet 3b04063d87be40f7ac1b46715d457983 (DEFAULT_TABLE table=rebalance_test_table_16 [id=bdae856e60144776aae709dddb837795]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:04.167801 6555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b04063d87be40f7ac1b46715d457983. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:04.168133 8454 tablet_bootstrap.cc:492] T 33b3834db0cd4407ae7c26e044459e87 P 3b164cbbc00f4e8eb3254ddd349a3ff9: No bootstrap required, opened a new log
I20240501 07:59:04.168224 8454 ts_tablet_manager.cc:1367] T 33b3834db0cd4407ae7c26e044459e87 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 07:59:04.168593 8455 tablet_bootstrap.cc:654] T 33b3834db0cd4407ae7c26e044459e87 P 8b8b3ad3aa8f4053b2167dd550f2086f: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:04.168720 8454 raft_consensus.cc:348] T 33b3834db0cd4407ae7c26e044459e87 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } }
I20240501 07:59:04.168841 8454 raft_consensus.cc:374] T 33b3834db0cd4407ae7c26e044459e87 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:04.168872 8454 raft_consensus.cc:729] T 33b3834db0cd4407ae7c26e044459e87 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b164cbbc00f4e8eb3254ddd349a3ff9, State: Initialized, Role: FOLLOWER
I20240501 07:59:04.169090 8454 consensus_queue.cc:260] T 33b3834db0cd4407ae7c26e044459e87 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } }
I20240501 07:59:04.169474 8454 ts_tablet_manager.cc:1398] T 33b3834db0cd4407ae7c26e044459e87 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 07:59:04.169582 8458 tablet_bootstrap.cc:654] T af8baea378be41f1806ea2cae61764bb P 1f9cb72be98941b5a8e504cbe34132b9: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:04.169692 8455 tablet_bootstrap.cc:492] T 33b3834db0cd4407ae7c26e044459e87 P 8b8b3ad3aa8f4053b2167dd550f2086f: No bootstrap required, opened a new log
I20240501 07:59:04.169771 8455 ts_tablet_manager.cc:1367] T 33b3834db0cd4407ae7c26e044459e87 P 8b8b3ad3aa8f4053b2167dd550f2086f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 07:59:04.170229 8455 raft_consensus.cc:348] T 33b3834db0cd4407ae7c26e044459e87 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } }
I20240501 07:59:04.170325 8455 raft_consensus.cc:374] T 33b3834db0cd4407ae7c26e044459e87 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:04.170351 8455 raft_consensus.cc:729] T 33b3834db0cd4407ae7c26e044459e87 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b8b3ad3aa8f4053b2167dd550f2086f, State: Initialized, Role: FOLLOWER
I20240501 07:59:04.170480 8455 consensus_queue.cc:260] T 33b3834db0cd4407ae7c26e044459e87 P 8b8b3ad3aa8f4053b2167dd550f2086f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } }
I20240501 07:59:04.170771 8455 ts_tablet_manager.cc:1398] T 33b3834db0cd4407ae7c26e044459e87 P 8b8b3ad3aa8f4053b2167dd550f2086f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:59:04.170941 8455 tablet_bootstrap.cc:492] T 3b04063d87be40f7ac1b46715d457983 P 8b8b3ad3aa8f4053b2167dd550f2086f: Bootstrap starting.
I20240501 07:59:04.171131 8454 tablet_bootstrap.cc:492] T 3b04063d87be40f7ac1b46715d457983 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Bootstrap starting.
I20240501 07:59:04.171558 8456 tablet_bootstrap.cc:492] T 33b3834db0cd4407ae7c26e044459e87 P 8315d22c56f7439a9e1fc82f153f018c: No bootstrap required, opened a new log
I20240501 07:59:04.171690 8456 ts_tablet_manager.cc:1367] T 33b3834db0cd4407ae7c26e044459e87 P 8315d22c56f7439a9e1fc82f153f018c: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 07:59:04.171949 8455 tablet_bootstrap.cc:654] T 3b04063d87be40f7ac1b46715d457983 P 8b8b3ad3aa8f4053b2167dd550f2086f: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:04.172294 8454 tablet_bootstrap.cc:654] T 3b04063d87be40f7ac1b46715d457983 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:04.172299 8456 raft_consensus.cc:348] T 33b3834db0cd4407ae7c26e044459e87 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } }
I20240501 07:59:04.172509 8456 raft_consensus.cc:374] T 33b3834db0cd4407ae7c26e044459e87 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:04.173346 8454 tablet_bootstrap.cc:492] T 3b04063d87be40f7ac1b46715d457983 P 3b164cbbc00f4e8eb3254ddd349a3ff9: No bootstrap required, opened a new log
I20240501 07:59:04.173429 8454 ts_tablet_manager.cc:1367] T 3b04063d87be40f7ac1b46715d457983 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240501 07:59:04.173905 8454 raft_consensus.cc:348] T 3b04063d87be40f7ac1b46715d457983 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } }
I20240501 07:59:04.174011 8454 raft_consensus.cc:374] T 3b04063d87be40f7ac1b46715d457983 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:04.174046 8454 raft_consensus.cc:729] T 3b04063d87be40f7ac1b46715d457983 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b164cbbc00f4e8eb3254ddd349a3ff9, State: Initialized, Role: FOLLOWER
I20240501 07:59:04.174455 8458 tablet_bootstrap.cc:492] T af8baea378be41f1806ea2cae61764bb P 1f9cb72be98941b5a8e504cbe34132b9: No bootstrap required, opened a new log
I20240501 07:59:04.174532 8455 tablet_bootstrap.cc:492] T 3b04063d87be40f7ac1b46715d457983 P 8b8b3ad3aa8f4053b2167dd550f2086f: No bootstrap required, opened a new log
I20240501 07:59:04.174543 8458 ts_tablet_manager.cc:1367] T af8baea378be41f1806ea2cae61764bb P 1f9cb72be98941b5a8e504cbe34132b9: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240501 07:59:04.174624 8455 ts_tablet_manager.cc:1367] T 3b04063d87be40f7ac1b46715d457983 P 8b8b3ad3aa8f4053b2167dd550f2086f: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 07:59:04.172585 8456 raft_consensus.cc:729] T 33b3834db0cd4407ae7c26e044459e87 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8315d22c56f7439a9e1fc82f153f018c, State: Initialized, Role: FOLLOWER
I20240501 07:59:04.174953 8454 consensus_queue.cc:260] T 3b04063d87be40f7ac1b46715d457983 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } }
I20240501 07:59:04.175107 8455 raft_consensus.cc:348] T 3b04063d87be40f7ac1b46715d457983 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } }
I20240501 07:59:04.175132 8456 consensus_queue.cc:260] T 33b3834db0cd4407ae7c26e044459e87 P 8315d22c56f7439a9e1fc82f153f018c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } }
I20240501 07:59:04.175228 8455 raft_consensus.cc:374] T 3b04063d87be40f7ac1b46715d457983 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:04.175302 8455 raft_consensus.cc:729] T 3b04063d87be40f7ac1b46715d457983 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b8b3ad3aa8f4053b2167dd550f2086f, State: Initialized, Role: FOLLOWER
I20240501 07:59:04.175367 8454 ts_tablet_manager.cc:1398] T 3b04063d87be40f7ac1b46715d457983 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 07:59:04.175432 8455 consensus_queue.cc:260] T 3b04063d87be40f7ac1b46715d457983 P 8b8b3ad3aa8f4053b2167dd550f2086f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } }
I20240501 07:59:04.175545 8456 ts_tablet_manager.cc:1398] T 33b3834db0cd4407ae7c26e044459e87 P 8315d22c56f7439a9e1fc82f153f018c: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240501 07:59:04.175721 8455 ts_tablet_manager.cc:1398] T 3b04063d87be40f7ac1b46715d457983 P 8b8b3ad3aa8f4053b2167dd550f2086f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:04.175662 8458 raft_consensus.cc:348] T af8baea378be41f1806ea2cae61764bb P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } }
I20240501 07:59:04.175770 8458 raft_consensus.cc:374] T af8baea378be41f1806ea2cae61764bb P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:04.175807 8458 raft_consensus.cc:729] T af8baea378be41f1806ea2cae61764bb P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f9cb72be98941b5a8e504cbe34132b9, State: Initialized, Role: FOLLOWER
I20240501 07:59:04.175858 8455 tablet_bootstrap.cc:492] T af8baea378be41f1806ea2cae61764bb P 8b8b3ad3aa8f4053b2167dd550f2086f: Bootstrap starting.
I20240501 07:59:04.176031 8458 consensus_queue.cc:260] T af8baea378be41f1806ea2cae61764bb P 1f9cb72be98941b5a8e504cbe34132b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } }
I20240501 07:59:04.176337 8458 ts_tablet_manager.cc:1398] T af8baea378be41f1806ea2cae61764bb P 1f9cb72be98941b5a8e504cbe34132b9: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:59:04.176533 8458 tablet_bootstrap.cc:492] T 3b04063d87be40f7ac1b46715d457983 P 1f9cb72be98941b5a8e504cbe34132b9: Bootstrap starting.
I20240501 07:59:04.176738 8455 tablet_bootstrap.cc:654] T af8baea378be41f1806ea2cae61764bb P 8b8b3ad3aa8f4053b2167dd550f2086f: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:04.177544 8458 tablet_bootstrap.cc:654] T 3b04063d87be40f7ac1b46715d457983 P 1f9cb72be98941b5a8e504cbe34132b9: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:04.179173 8458 tablet_bootstrap.cc:492] T 3b04063d87be40f7ac1b46715d457983 P 1f9cb72be98941b5a8e504cbe34132b9: No bootstrap required, opened a new log
I20240501 07:59:04.179251 8458 ts_tablet_manager.cc:1367] T 3b04063d87be40f7ac1b46715d457983 P 1f9cb72be98941b5a8e504cbe34132b9: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240501 07:59:04.179687 8455 tablet_bootstrap.cc:492] T af8baea378be41f1806ea2cae61764bb P 8b8b3ad3aa8f4053b2167dd550f2086f: No bootstrap required, opened a new log
I20240501 07:59:04.179661 8458 raft_consensus.cc:348] T 3b04063d87be40f7ac1b46715d457983 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } }
I20240501 07:59:04.179762 8458 raft_consensus.cc:374] T 3b04063d87be40f7ac1b46715d457983 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:04.179772 8455 ts_tablet_manager.cc:1367] T af8baea378be41f1806ea2cae61764bb P 8b8b3ad3aa8f4053b2167dd550f2086f: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20240501 07:59:04.179795 8458 raft_consensus.cc:729] T 3b04063d87be40f7ac1b46715d457983 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f9cb72be98941b5a8e504cbe34132b9, State: Initialized, Role: FOLLOWER
I20240501 07:59:04.179904 8458 consensus_queue.cc:260] T 3b04063d87be40f7ac1b46715d457983 P 1f9cb72be98941b5a8e504cbe34132b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } }
I20240501 07:59:04.180212 8458 ts_tablet_manager.cc:1398] T 3b04063d87be40f7ac1b46715d457983 P 1f9cb72be98941b5a8e504cbe34132b9: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:59:04.180210 8455 raft_consensus.cc:348] T af8baea378be41f1806ea2cae61764bb P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } }
I20240501 07:59:04.180325 8455 raft_consensus.cc:374] T af8baea378be41f1806ea2cae61764bb P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:04.180354 8455 raft_consensus.cc:729] T af8baea378be41f1806ea2cae61764bb P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b8b3ad3aa8f4053b2167dd550f2086f, State: Initialized, Role: FOLLOWER
I20240501 07:59:04.180475 8455 consensus_queue.cc:260] T af8baea378be41f1806ea2cae61764bb P 8b8b3ad3aa8f4053b2167dd550f2086f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } }
I20240501 07:59:04.175690 8456 tablet_bootstrap.cc:492] T af8baea378be41f1806ea2cae61764bb P 8315d22c56f7439a9e1fc82f153f018c: Bootstrap starting.
I20240501 07:59:04.180747 8455 ts_tablet_manager.cc:1398] T af8baea378be41f1806ea2cae61764bb P 8b8b3ad3aa8f4053b2167dd550f2086f: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 07:59:04.182188 8456 tablet_bootstrap.cc:654] T af8baea378be41f1806ea2cae61764bb P 8315d22c56f7439a9e1fc82f153f018c: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:04.183372 8456 tablet_bootstrap.cc:492] T af8baea378be41f1806ea2cae61764bb P 8315d22c56f7439a9e1fc82f153f018c: No bootstrap required, opened a new log
I20240501 07:59:04.183457 8456 ts_tablet_manager.cc:1367] T af8baea378be41f1806ea2cae61764bb P 8315d22c56f7439a9e1fc82f153f018c: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240501 07:59:04.183813 8456 raft_consensus.cc:348] T af8baea378be41f1806ea2cae61764bb P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } }
I20240501 07:59:04.183904 8456 raft_consensus.cc:374] T af8baea378be41f1806ea2cae61764bb P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:04.183940 8456 raft_consensus.cc:729] T af8baea378be41f1806ea2cae61764bb P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8315d22c56f7439a9e1fc82f153f018c, State: Initialized, Role: FOLLOWER
I20240501 07:59:04.184077 8456 consensus_queue.cc:260] T af8baea378be41f1806ea2cae61764bb P 8315d22c56f7439a9e1fc82f153f018c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } }
I20240501 07:59:04.184361 8456 ts_tablet_manager.cc:1398] T af8baea378be41f1806ea2cae61764bb P 8315d22c56f7439a9e1fc82f153f018c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:59:04.188810 8166 raft_consensus.cc:482] T 33b3834db0cd4407ae7c26e044459e87 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:04.188915 8166 raft_consensus.cc:504] T 33b3834db0cd4407ae7c26e044459e87 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } }
I20240501 07:59:04.189339 8166 leader_election.cc:288] T 33b3834db0cd4407ae7c26e044459e87 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129:46875), 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130:32941)
I20240501 07:59:04.189685 6313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33b3834db0cd4407ae7c26e044459e87" candidate_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" is_pre_election: true
I20240501 07:59:04.189743 6445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33b3834db0cd4407ae7c26e044459e87" candidate_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8315d22c56f7439a9e1fc82f153f018c" is_pre_election: true
I20240501 07:59:04.189834 6313 raft_consensus.cc:2454] T 33b3834db0cd4407ae7c26e044459e87 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b164cbbc00f4e8eb3254ddd349a3ff9 in term 0.
I20240501 07:59:04.189863 6445 raft_consensus.cc:2454] T 33b3834db0cd4407ae7c26e044459e87 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b164cbbc00f4e8eb3254ddd349a3ff9 in term 0.
I20240501 07:59:04.190132 6512 leader_election.cc:302] T 33b3834db0cd4407ae7c26e044459e87 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b164cbbc00f4e8eb3254ddd349a3ff9, 8315d22c56f7439a9e1fc82f153f018c; no voters:
I20240501 07:59:04.190352 8166 raft_consensus.cc:2781] T 33b3834db0cd4407ae7c26e044459e87 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:04.190398 8166 raft_consensus.cc:482] T 33b3834db0cd4407ae7c26e044459e87 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:04.190423 8166 raft_consensus.cc:3037] T 33b3834db0cd4407ae7c26e044459e87 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:04.191323 8166 raft_consensus.cc:504] T 33b3834db0cd4407ae7c26e044459e87 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } }
I20240501 07:59:04.191637 8166 leader_election.cc:288] T 33b3834db0cd4407ae7c26e044459e87 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [CANDIDATE]: Term 1 election: Requested vote from peers 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129:46875), 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130:32941)
I20240501 07:59:04.191929 6313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33b3834db0cd4407ae7c26e044459e87" candidate_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f"
I20240501 07:59:04.192059 6313 raft_consensus.cc:3037] T 33b3834db0cd4407ae7c26e044459e87 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:04.192788 6313 raft_consensus.cc:2454] T 33b3834db0cd4407ae7c26e044459e87 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b164cbbc00f4e8eb3254ddd349a3ff9 in term 1.
I20240501 07:59:04.193048 6445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33b3834db0cd4407ae7c26e044459e87" candidate_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8315d22c56f7439a9e1fc82f153f018c"
I20240501 07:59:04.193166 6445 raft_consensus.cc:3037] T 33b3834db0cd4407ae7c26e044459e87 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:04.193331 6510 leader_election.cc:302] T 33b3834db0cd4407ae7c26e044459e87 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b164cbbc00f4e8eb3254ddd349a3ff9, 8b8b3ad3aa8f4053b2167dd550f2086f; no voters:
I20240501 07:59:04.193493 8166 raft_consensus.cc:2781] T 33b3834db0cd4407ae7c26e044459e87 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:04.193555 8166 raft_consensus.cc:686] T 33b3834db0cd4407ae7c26e044459e87 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 LEADER]: Becoming Leader. State: Replica: 3b164cbbc00f4e8eb3254ddd349a3ff9, State: Running, Role: LEADER
I20240501 07:59:04.193682 8166 consensus_queue.cc:237] T 33b3834db0cd4407ae7c26e044459e87 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } }
I20240501 07:59:04.194103 6445 raft_consensus.cc:2454] T 33b3834db0cd4407ae7c26e044459e87 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b164cbbc00f4e8eb3254ddd349a3ff9 in term 1.
I20240501 07:59:04.195577 6167 catalog_manager.cc:5484] T 33b3834db0cd4407ae7c26e044459e87 P 3b164cbbc00f4e8eb3254ddd349a3ff9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } health_report { overall_health: UNKNOWN } } }
I20240501 07:59:04.254293 8166 raft_consensus.cc:482] T 3b04063d87be40f7ac1b46715d457983 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:04.254477 8166 raft_consensus.cc:504] T 3b04063d87be40f7ac1b46715d457983 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } }
I20240501 07:59:04.254925 8166 leader_election.cc:288] T 3b04063d87be40f7ac1b46715d457983 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132:36773), 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129:46875)
I20240501 07:59:04.255291 6708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b04063d87be40f7ac1b46715d457983" candidate_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f9cb72be98941b5a8e504cbe34132b9" is_pre_election: true
I20240501 07:59:04.255432 6708 raft_consensus.cc:2454] T 3b04063d87be40f7ac1b46715d457983 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b164cbbc00f4e8eb3254ddd349a3ff9 in term 0.
I20240501 07:59:04.255715 6314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b04063d87be40f7ac1b46715d457983" candidate_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" is_pre_election: true
I20240501 07:59:04.255827 6512 leader_election.cc:302] T 3b04063d87be40f7ac1b46715d457983 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f9cb72be98941b5a8e504cbe34132b9, 3b164cbbc00f4e8eb3254ddd349a3ff9; no voters:
I20240501 07:59:04.255847 6314 raft_consensus.cc:2454] T 3b04063d87be40f7ac1b46715d457983 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b164cbbc00f4e8eb3254ddd349a3ff9 in term 0.
I20240501 07:59:04.256017 8468 raft_consensus.cc:2781] T 3b04063d87be40f7ac1b46715d457983 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:04.256075 8468 raft_consensus.cc:482] T 3b04063d87be40f7ac1b46715d457983 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:04.256134 8468 raft_consensus.cc:3037] T 3b04063d87be40f7ac1b46715d457983 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:04.257038 8468 raft_consensus.cc:504] T 3b04063d87be40f7ac1b46715d457983 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } }
I20240501 07:59:04.257334 8468 leader_election.cc:288] T 3b04063d87be40f7ac1b46715d457983 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [CANDIDATE]: Term 1 election: Requested vote from peers 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132:36773), 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129:46875)
I20240501 07:59:04.257625 6708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b04063d87be40f7ac1b46715d457983" candidate_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f9cb72be98941b5a8e504cbe34132b9"
I20240501 07:59:04.257632 6314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b04063d87be40f7ac1b46715d457983" candidate_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f"
I20240501 07:59:04.257719 6708 raft_consensus.cc:3037] T 3b04063d87be40f7ac1b46715d457983 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:04.257740 6314 raft_consensus.cc:3037] T 3b04063d87be40f7ac1b46715d457983 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:04.258450 6708 raft_consensus.cc:2454] T 3b04063d87be40f7ac1b46715d457983 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b164cbbc00f4e8eb3254ddd349a3ff9 in term 1.
I20240501 07:59:04.258530 6314 raft_consensus.cc:2454] T 3b04063d87be40f7ac1b46715d457983 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b164cbbc00f4e8eb3254ddd349a3ff9 in term 1.
I20240501 07:59:04.258718 6512 leader_election.cc:302] T 3b04063d87be40f7ac1b46715d457983 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f9cb72be98941b5a8e504cbe34132b9, 3b164cbbc00f4e8eb3254ddd349a3ff9; no voters:
I20240501 07:59:04.258926 8468 raft_consensus.cc:2781] T 3b04063d87be40f7ac1b46715d457983 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:04.259032 8468 raft_consensus.cc:686] T 3b04063d87be40f7ac1b46715d457983 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 LEADER]: Becoming Leader. State: Replica: 3b164cbbc00f4e8eb3254ddd349a3ff9, State: Running, Role: LEADER
I20240501 07:59:04.259174 8468 consensus_queue.cc:237] T 3b04063d87be40f7ac1b46715d457983 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } }
I20240501 07:59:04.261504 6167 catalog_manager.cc:5484] T 3b04063d87be40f7ac1b46715d457983 P 3b164cbbc00f4e8eb3254ddd349a3ff9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } health_report { overall_health: UNKNOWN } } }
I20240501 07:59:04.465176 8385 raft_consensus.cc:482] T af8baea378be41f1806ea2cae61764bb P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:04.465328 8385 raft_consensus.cc:504] T af8baea378be41f1806ea2cae61764bb P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } }
I20240501 07:59:04.465780 8385 leader_election.cc:288] T af8baea378be41f1806ea2cae61764bb P 1f9cb72be98941b5a8e504cbe34132b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129:46875), 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130:32941)
I20240501 07:59:04.466217 6314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af8baea378be41f1806ea2cae61764bb" candidate_uuid: "1f9cb72be98941b5a8e504cbe34132b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" is_pre_election: true
I20240501 07:59:04.466238 6445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af8baea378be41f1806ea2cae61764bb" candidate_uuid: "1f9cb72be98941b5a8e504cbe34132b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8315d22c56f7439a9e1fc82f153f018c" is_pre_election: true
I20240501 07:59:04.466368 6445 raft_consensus.cc:2454] T af8baea378be41f1806ea2cae61764bb P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f9cb72be98941b5a8e504cbe34132b9 in term 0.
I20240501 07:59:04.466367 6314 raft_consensus.cc:2454] T af8baea378be41f1806ea2cae61764bb P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f9cb72be98941b5a8e504cbe34132b9 in term 0.
I20240501 07:59:04.466692 6643 leader_election.cc:302] T af8baea378be41f1806ea2cae61764bb P 1f9cb72be98941b5a8e504cbe34132b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f9cb72be98941b5a8e504cbe34132b9, 8315d22c56f7439a9e1fc82f153f018c; no voters:
I20240501 07:59:04.466869 8385 raft_consensus.cc:2781] T af8baea378be41f1806ea2cae61764bb P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:04.466930 8385 raft_consensus.cc:482] T af8baea378be41f1806ea2cae61764bb P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:04.466974 8385 raft_consensus.cc:3037] T af8baea378be41f1806ea2cae61764bb P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:04.468215 8385 raft_consensus.cc:504] T af8baea378be41f1806ea2cae61764bb P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } }
I20240501 07:59:04.468596 8385 leader_election.cc:288] T af8baea378be41f1806ea2cae61764bb P 1f9cb72be98941b5a8e504cbe34132b9 [CANDIDATE]: Term 1 election: Requested vote from peers 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129:46875), 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130:32941)
I20240501 07:59:04.469012 6314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af8baea378be41f1806ea2cae61764bb" candidate_uuid: "1f9cb72be98941b5a8e504cbe34132b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f"
I20240501 07:59:04.469142 6314 raft_consensus.cc:3037] T af8baea378be41f1806ea2cae61764bb P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:04.469126 6445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af8baea378be41f1806ea2cae61764bb" candidate_uuid: "1f9cb72be98941b5a8e504cbe34132b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8315d22c56f7439a9e1fc82f153f018c"
I20240501 07:59:04.469240 6445 raft_consensus.cc:3037] T af8baea378be41f1806ea2cae61764bb P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:04.469951 6314 raft_consensus.cc:2454] T af8baea378be41f1806ea2cae61764bb P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f9cb72be98941b5a8e504cbe34132b9 in term 1.
I20240501 07:59:04.470197 6445 raft_consensus.cc:2454] T af8baea378be41f1806ea2cae61764bb P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f9cb72be98941b5a8e504cbe34132b9 in term 1.
I20240501 07:59:04.470261 6641 leader_election.cc:302] T af8baea378be41f1806ea2cae61764bb P 1f9cb72be98941b5a8e504cbe34132b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f9cb72be98941b5a8e504cbe34132b9, 8b8b3ad3aa8f4053b2167dd550f2086f; no voters:
I20240501 07:59:04.470422 8385 raft_consensus.cc:2781] T af8baea378be41f1806ea2cae61764bb P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:04.470518 8385 raft_consensus.cc:686] T af8baea378be41f1806ea2cae61764bb P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 LEADER]: Becoming Leader. State: Replica: 1f9cb72be98941b5a8e504cbe34132b9, State: Running, Role: LEADER
I20240501 07:59:04.470674 8385 consensus_queue.cc:237] T af8baea378be41f1806ea2cae61764bb P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } }
I20240501 07:59:04.472790 6167 catalog_manager.cc:5484] T af8baea378be41f1806ea2cae61764bb P 1f9cb72be98941b5a8e504cbe34132b9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132). New cstate: current_term: 1 leader_uuid: "1f9cb72be98941b5a8e504cbe34132b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } health_report { overall_health: UNKNOWN } } }
I20240501 07:59:04.576180 8468 consensus_queue.cc:1035] T 33b3834db0cd4407ae7c26e044459e87 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:04.580801 8468 consensus_queue.cc:1035] T 33b3834db0cd4407ae7c26e044459e87 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 07:59:04.659909 8471 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:59:04.660189 8471 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240501 07:59:04.663374 8166 consensus_queue.cc:1035] T 3b04063d87be40f7ac1b46715d457983 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 07:59:04.663920 8471 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 07:59:04.667119 8235 consensus_queue.cc:1035] T 3b04063d87be40f7ac1b46715d457983 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_16'
I20240501 07:59:04.747979 6445 raft_consensus.cc:1261] T af8baea378be41f1806ea2cae61764bb P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Refusing update from remote peer 1f9cb72be98941b5a8e504cbe34132b9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:04.748119 6314 raft_consensus.cc:1261] T af8baea378be41f1806ea2cae61764bb P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Refusing update from remote peer 1f9cb72be98941b5a8e504cbe34132b9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:04.748531 8385 consensus_queue.cc:1035] T af8baea378be41f1806ea2cae61764bb P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:04.751507 8385 consensus_queue.cc:1035] T af8baea378be41f1806ea2cae61764bb P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 104.733 ms
time per row: 0.0523667 ms
I20240501 07:59:04.815311 6167 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55950:
name: "rebalance_test_table_17"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 07:59:04.821311 6425 tablet_service.cc:1467] Processing CreateTablet for tablet dc550f54e8024d47b1955905e7858a1e (DEFAULT_TABLE table=rebalance_test_table_17 [id=4de37f887eb14e2dbe5d1609df1e538f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:04.821681 6425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc550f54e8024d47b1955905e7858a1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:04.821633 6424 tablet_service.cc:1467] Processing CreateTablet for tablet 6e2ac8c8857d45b88d3b51147d2609aa (DEFAULT_TABLE table=rebalance_test_table_17 [id=4de37f887eb14e2dbe5d1609df1e538f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:04.821830 6424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e2ac8c8857d45b88d3b51147d2609aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:04.821844 6686 tablet_service.cc:1467] Processing CreateTablet for tablet 7dd6566ccb6e43388146a6dfd1c4a88c (DEFAULT_TABLE table=rebalance_test_table_17 [id=4de37f887eb14e2dbe5d1609df1e538f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:04.822125 6686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7dd6566ccb6e43388146a6dfd1c4a88c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:04.823804 6293 tablet_service.cc:1467] Processing CreateTablet for tablet 6e2ac8c8857d45b88d3b51147d2609aa (DEFAULT_TABLE table=rebalance_test_table_17 [id=4de37f887eb14e2dbe5d1609df1e538f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:04.824084 6293 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e2ac8c8857d45b88d3b51147d2609aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:04.824164 6687 tablet_service.cc:1467] Processing CreateTablet for tablet dc550f54e8024d47b1955905e7858a1e (DEFAULT_TABLE table=rebalance_test_table_17 [id=4de37f887eb14e2dbe5d1609df1e538f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:04.824383 6687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc550f54e8024d47b1955905e7858a1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:04.825309 6426 tablet_service.cc:1467] Processing CreateTablet for tablet 7dd6566ccb6e43388146a6dfd1c4a88c (DEFAULT_TABLE table=rebalance_test_table_17 [id=4de37f887eb14e2dbe5d1609df1e538f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:04.825441 6555 tablet_service.cc:1467] Processing CreateTablet for tablet 7dd6566ccb6e43388146a6dfd1c4a88c (DEFAULT_TABLE table=rebalance_test_table_17 [id=4de37f887eb14e2dbe5d1609df1e538f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:04.825551 6426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7dd6566ccb6e43388146a6dfd1c4a88c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:04.825692 6555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7dd6566ccb6e43388146a6dfd1c4a88c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:04.826473 6557 tablet_service.cc:1467] Processing CreateTablet for tablet 6e2ac8c8857d45b88d3b51147d2609aa (DEFAULT_TABLE table=rebalance_test_table_17 [id=4de37f887eb14e2dbe5d1609df1e538f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:04.826532 6556 tablet_service.cc:1467] Processing CreateTablet for tablet dc550f54e8024d47b1955905e7858a1e (DEFAULT_TABLE table=rebalance_test_table_17 [id=4de37f887eb14e2dbe5d1609df1e538f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:04.826707 6557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e2ac8c8857d45b88d3b51147d2609aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:04.829074 8533 tablet_bootstrap.cc:492] T 6e2ac8c8857d45b88d3b51147d2609aa P 8315d22c56f7439a9e1fc82f153f018c: Bootstrap starting.
I20240501 07:59:04.830222 6556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc550f54e8024d47b1955905e7858a1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:04.830590 8534 tablet_bootstrap.cc:492] T 7dd6566ccb6e43388146a6dfd1c4a88c P 1f9cb72be98941b5a8e504cbe34132b9: Bootstrap starting.
I20240501 07:59:04.830873 8533 tablet_bootstrap.cc:654] T 6e2ac8c8857d45b88d3b51147d2609aa P 8315d22c56f7439a9e1fc82f153f018c: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:04.831516 8534 tablet_bootstrap.cc:654] T 7dd6566ccb6e43388146a6dfd1c4a88c P 1f9cb72be98941b5a8e504cbe34132b9: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:04.832125 8533 tablet_bootstrap.cc:492] T 6e2ac8c8857d45b88d3b51147d2609aa P 8315d22c56f7439a9e1fc82f153f018c: No bootstrap required, opened a new log
I20240501 07:59:04.832211 8533 ts_tablet_manager.cc:1367] T 6e2ac8c8857d45b88d3b51147d2609aa P 8315d22c56f7439a9e1fc82f153f018c: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 07:59:04.832221 8535 tablet_bootstrap.cc:492] T 6e2ac8c8857d45b88d3b51147d2609aa P 8b8b3ad3aa8f4053b2167dd550f2086f: Bootstrap starting.
I20240501 07:59:04.832751 8533 raft_consensus.cc:348] T 6e2ac8c8857d45b88d3b51147d2609aa P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } }
I20240501 07:59:04.832856 8533 raft_consensus.cc:374] T 6e2ac8c8857d45b88d3b51147d2609aa P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:04.832887 8533 raft_consensus.cc:729] T 6e2ac8c8857d45b88d3b51147d2609aa P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8315d22c56f7439a9e1fc82f153f018c, State: Initialized, Role: FOLLOWER
I20240501 07:59:04.833024 8533 consensus_queue.cc:260] T 6e2ac8c8857d45b88d3b51147d2609aa P 8315d22c56f7439a9e1fc82f153f018c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } }
I20240501 07:59:04.833225 8535 tablet_bootstrap.cc:654] T 6e2ac8c8857d45b88d3b51147d2609aa P 8b8b3ad3aa8f4053b2167dd550f2086f: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:04.833326 8533 ts_tablet_manager.cc:1398] T 6e2ac8c8857d45b88d3b51147d2609aa P 8315d22c56f7439a9e1fc82f153f018c: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 07:59:04.833515 8533 tablet_bootstrap.cc:492] T 7dd6566ccb6e43388146a6dfd1c4a88c P 8315d22c56f7439a9e1fc82f153f018c: Bootstrap starting.
I20240501 07:59:04.833925 8536 tablet_bootstrap.cc:492] T 7dd6566ccb6e43388146a6dfd1c4a88c P 3b164cbbc00f4e8eb3254ddd349a3ff9: Bootstrap starting.
I20240501 07:59:04.834350 8533 tablet_bootstrap.cc:654] T 7dd6566ccb6e43388146a6dfd1c4a88c P 8315d22c56f7439a9e1fc82f153f018c: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:04.834865 8536 tablet_bootstrap.cc:654] T 7dd6566ccb6e43388146a6dfd1c4a88c P 3b164cbbc00f4e8eb3254ddd349a3ff9: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:04.835961 8533 tablet_bootstrap.cc:492] T 7dd6566ccb6e43388146a6dfd1c4a88c P 8315d22c56f7439a9e1fc82f153f018c: No bootstrap required, opened a new log
I20240501 07:59:04.835968 8534 tablet_bootstrap.cc:492] T 7dd6566ccb6e43388146a6dfd1c4a88c P 1f9cb72be98941b5a8e504cbe34132b9: No bootstrap required, opened a new log
I20240501 07:59:04.836047 8533 ts_tablet_manager.cc:1367] T 7dd6566ccb6e43388146a6dfd1c4a88c P 8315d22c56f7439a9e1fc82f153f018c: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20240501 07:59:04.836057 8534 ts_tablet_manager.cc:1367] T 7dd6566ccb6e43388146a6dfd1c4a88c P 1f9cb72be98941b5a8e504cbe34132b9: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 07:59:04.836481 8533 raft_consensus.cc:348] T 7dd6566ccb6e43388146a6dfd1c4a88c P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } }
I20240501 07:59:04.836498 8534 raft_consensus.cc:348] T 7dd6566ccb6e43388146a6dfd1c4a88c P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } }
I20240501 07:59:04.836588 8533 raft_consensus.cc:374] T 7dd6566ccb6e43388146a6dfd1c4a88c P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:04.836591 8534 raft_consensus.cc:374] T 7dd6566ccb6e43388146a6dfd1c4a88c P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:04.836623 8533 raft_consensus.cc:729] T 7dd6566ccb6e43388146a6dfd1c4a88c P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8315d22c56f7439a9e1fc82f153f018c, State: Initialized, Role: FOLLOWER
I20240501 07:59:04.836625 8534 raft_consensus.cc:729] T 7dd6566ccb6e43388146a6dfd1c4a88c P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f9cb72be98941b5a8e504cbe34132b9, State: Initialized, Role: FOLLOWER
I20240501 07:59:04.836760 8533 consensus_queue.cc:260] T 7dd6566ccb6e43388146a6dfd1c4a88c P 8315d22c56f7439a9e1fc82f153f018c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } }
I20240501 07:59:04.836802 8534 consensus_queue.cc:260] T 7dd6566ccb6e43388146a6dfd1c4a88c P 1f9cb72be98941b5a8e504cbe34132b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } }
I20240501 07:59:04.837080 8533 ts_tablet_manager.cc:1398] T 7dd6566ccb6e43388146a6dfd1c4a88c P 8315d22c56f7439a9e1fc82f153f018c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:04.837246 8533 tablet_bootstrap.cc:492] T dc550f54e8024d47b1955905e7858a1e P 8315d22c56f7439a9e1fc82f153f018c: Bootstrap starting.
I20240501 07:59:04.838127 8533 tablet_bootstrap.cc:654] T dc550f54e8024d47b1955905e7858a1e P 8315d22c56f7439a9e1fc82f153f018c: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:04.839427 8534 ts_tablet_manager.cc:1398] T 7dd6566ccb6e43388146a6dfd1c4a88c P 1f9cb72be98941b5a8e504cbe34132b9: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 07:59:04.839818 8533 tablet_bootstrap.cc:492] T dc550f54e8024d47b1955905e7858a1e P 8315d22c56f7439a9e1fc82f153f018c: No bootstrap required, opened a new log
I20240501 07:59:04.839887 8533 ts_tablet_manager.cc:1367] T dc550f54e8024d47b1955905e7858a1e P 8315d22c56f7439a9e1fc82f153f018c: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240501 07:59:04.840402 8533 raft_consensus.cc:348] T dc550f54e8024d47b1955905e7858a1e P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:59:04.840494 8533 raft_consensus.cc:374] T dc550f54e8024d47b1955905e7858a1e P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:04.840529 8533 raft_consensus.cc:729] T dc550f54e8024d47b1955905e7858a1e P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8315d22c56f7439a9e1fc82f153f018c, State: Initialized, Role: FOLLOWER
I20240501 07:59:04.840642 8533 consensus_queue.cc:260] T dc550f54e8024d47b1955905e7858a1e P 8315d22c56f7439a9e1fc82f153f018c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:59:04.840916 8533 ts_tablet_manager.cc:1398] T dc550f54e8024d47b1955905e7858a1e P 8315d22c56f7439a9e1fc82f153f018c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:04.841596 8534 tablet_bootstrap.cc:492] T dc550f54e8024d47b1955905e7858a1e P 1f9cb72be98941b5a8e504cbe34132b9: Bootstrap starting.
I20240501 07:59:04.842541 8534 tablet_bootstrap.cc:654] T dc550f54e8024d47b1955905e7858a1e P 1f9cb72be98941b5a8e504cbe34132b9: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:04.843613 8535 tablet_bootstrap.cc:492] T 6e2ac8c8857d45b88d3b51147d2609aa P 8b8b3ad3aa8f4053b2167dd550f2086f: No bootstrap required, opened a new log
I20240501 07:59:04.843710 8535 ts_tablet_manager.cc:1367] T 6e2ac8c8857d45b88d3b51147d2609aa P 8b8b3ad3aa8f4053b2167dd550f2086f: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240501 07:59:04.843851 8534 tablet_bootstrap.cc:492] T dc550f54e8024d47b1955905e7858a1e P 1f9cb72be98941b5a8e504cbe34132b9: No bootstrap required, opened a new log
I20240501 07:59:04.843961 8534 ts_tablet_manager.cc:1367] T dc550f54e8024d47b1955905e7858a1e P 1f9cb72be98941b5a8e504cbe34132b9: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 07:59:04.844215 8535 raft_consensus.cc:348] T 6e2ac8c8857d45b88d3b51147d2609aa P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } }
I20240501 07:59:04.844333 8535 raft_consensus.cc:374] T 6e2ac8c8857d45b88d3b51147d2609aa P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:04.844370 8535 raft_consensus.cc:729] T 6e2ac8c8857d45b88d3b51147d2609aa P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b8b3ad3aa8f4053b2167dd550f2086f, State: Initialized, Role: FOLLOWER
I20240501 07:59:04.844511 8535 consensus_queue.cc:260] T 6e2ac8c8857d45b88d3b51147d2609aa P 8b8b3ad3aa8f4053b2167dd550f2086f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } }
I20240501 07:59:04.844712 8534 raft_consensus.cc:348] T dc550f54e8024d47b1955905e7858a1e P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:59:04.844811 8535 ts_tablet_manager.cc:1398] T 6e2ac8c8857d45b88d3b51147d2609aa P 8b8b3ad3aa8f4053b2167dd550f2086f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:59:04.844889 8534 raft_consensus.cc:374] T dc550f54e8024d47b1955905e7858a1e P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:04.844997 8534 raft_consensus.cc:729] T dc550f54e8024d47b1955905e7858a1e P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f9cb72be98941b5a8e504cbe34132b9, State: Initialized, Role: FOLLOWER
I20240501 07:59:04.845206 8534 consensus_queue.cc:260] T dc550f54e8024d47b1955905e7858a1e P 1f9cb72be98941b5a8e504cbe34132b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:59:04.845712 8534 ts_tablet_manager.cc:1398] T dc550f54e8024d47b1955905e7858a1e P 1f9cb72be98941b5a8e504cbe34132b9: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:59:04.846410 8536 tablet_bootstrap.cc:492] T 7dd6566ccb6e43388146a6dfd1c4a88c P 3b164cbbc00f4e8eb3254ddd349a3ff9: No bootstrap required, opened a new log
I20240501 07:59:04.846551 8536 ts_tablet_manager.cc:1367] T 7dd6566ccb6e43388146a6dfd1c4a88c P 3b164cbbc00f4e8eb3254ddd349a3ff9: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20240501 07:59:04.847708 8536 raft_consensus.cc:348] T 7dd6566ccb6e43388146a6dfd1c4a88c P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } }
I20240501 07:59:04.847899 8536 raft_consensus.cc:374] T 7dd6566ccb6e43388146a6dfd1c4a88c P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:04.848116 8536 raft_consensus.cc:729] T 7dd6566ccb6e43388146a6dfd1c4a88c P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b164cbbc00f4e8eb3254ddd349a3ff9, State: Initialized, Role: FOLLOWER
I20240501 07:59:04.848317 8536 consensus_queue.cc:260] T 7dd6566ccb6e43388146a6dfd1c4a88c P 3b164cbbc00f4e8eb3254ddd349a3ff9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } }
I20240501 07:59:04.848695 8536 ts_tablet_manager.cc:1398] T 7dd6566ccb6e43388146a6dfd1c4a88c P 3b164cbbc00f4e8eb3254ddd349a3ff9: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:59:04.848977 8536 tablet_bootstrap.cc:492] T 6e2ac8c8857d45b88d3b51147d2609aa P 3b164cbbc00f4e8eb3254ddd349a3ff9: Bootstrap starting.
I20240501 07:59:04.850513 8536 tablet_bootstrap.cc:654] T 6e2ac8c8857d45b88d3b51147d2609aa P 3b164cbbc00f4e8eb3254ddd349a3ff9: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:04.851686 8536 tablet_bootstrap.cc:492] T 6e2ac8c8857d45b88d3b51147d2609aa P 3b164cbbc00f4e8eb3254ddd349a3ff9: No bootstrap required, opened a new log
I20240501 07:59:04.851763 8536 ts_tablet_manager.cc:1367] T 6e2ac8c8857d45b88d3b51147d2609aa P 3b164cbbc00f4e8eb3254ddd349a3ff9: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 07:59:04.852167 8536 raft_consensus.cc:348] T 6e2ac8c8857d45b88d3b51147d2609aa P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } }
I20240501 07:59:04.852273 8536 raft_consensus.cc:374] T 6e2ac8c8857d45b88d3b51147d2609aa P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:04.852303 8536 raft_consensus.cc:729] T 6e2ac8c8857d45b88d3b51147d2609aa P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b164cbbc00f4e8eb3254ddd349a3ff9, State: Initialized, Role: FOLLOWER
I20240501 07:59:04.852429 8536 consensus_queue.cc:260] T 6e2ac8c8857d45b88d3b51147d2609aa P 3b164cbbc00f4e8eb3254ddd349a3ff9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } }
I20240501 07:59:04.852676 8536 ts_tablet_manager.cc:1398] T 6e2ac8c8857d45b88d3b51147d2609aa P 3b164cbbc00f4e8eb3254ddd349a3ff9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:04.853107 8536 tablet_bootstrap.cc:492] T dc550f54e8024d47b1955905e7858a1e P 3b164cbbc00f4e8eb3254ddd349a3ff9: Bootstrap starting.
I20240501 07:59:04.853993 8536 tablet_bootstrap.cc:654] T dc550f54e8024d47b1955905e7858a1e P 3b164cbbc00f4e8eb3254ddd349a3ff9: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:04.855332 8536 tablet_bootstrap.cc:492] T dc550f54e8024d47b1955905e7858a1e P 3b164cbbc00f4e8eb3254ddd349a3ff9: No bootstrap required, opened a new log
I20240501 07:59:04.855410 8536 ts_tablet_manager.cc:1367] T dc550f54e8024d47b1955905e7858a1e P 3b164cbbc00f4e8eb3254ddd349a3ff9: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 07:59:04.855798 8536 raft_consensus.cc:348] T dc550f54e8024d47b1955905e7858a1e P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:59:04.855907 8536 raft_consensus.cc:374] T dc550f54e8024d47b1955905e7858a1e P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:04.855942 8536 raft_consensus.cc:729] T dc550f54e8024d47b1955905e7858a1e P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b164cbbc00f4e8eb3254ddd349a3ff9, State: Initialized, Role: FOLLOWER
I20240501 07:59:04.856056 8536 consensus_queue.cc:260] T dc550f54e8024d47b1955905e7858a1e P 3b164cbbc00f4e8eb3254ddd349a3ff9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:59:04.856289 8536 ts_tablet_manager.cc:1398] T dc550f54e8024d47b1955905e7858a1e P 3b164cbbc00f4e8eb3254ddd349a3ff9: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:59:04.894734 8520 raft_consensus.cc:482] T 6e2ac8c8857d45b88d3b51147d2609aa P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:04.894872 8520 raft_consensus.cc:504] T 6e2ac8c8857d45b88d3b51147d2609aa P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } }
I20240501 07:59:04.895349 8520 leader_election.cc:288] T 6e2ac8c8857d45b88d3b51147d2609aa P 3b164cbbc00f4e8eb3254ddd349a3ff9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129:46875), 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130:32941)
I20240501 07:59:04.895637 6313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e2ac8c8857d45b88d3b51147d2609aa" candidate_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" is_pre_election: true
I20240501 07:59:04.895784 6313 raft_consensus.cc:2454] T 6e2ac8c8857d45b88d3b51147d2609aa P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b164cbbc00f4e8eb3254ddd349a3ff9 in term 0.
I20240501 07:59:04.895862 6444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e2ac8c8857d45b88d3b51147d2609aa" candidate_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8315d22c56f7439a9e1fc82f153f018c" is_pre_election: true
I20240501 07:59:04.896036 6444 raft_consensus.cc:2454] T 6e2ac8c8857d45b88d3b51147d2609aa P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b164cbbc00f4e8eb3254ddd349a3ff9 in term 0.
I20240501 07:59:04.896083 6510 leader_election.cc:302] T 6e2ac8c8857d45b88d3b51147d2609aa P 3b164cbbc00f4e8eb3254ddd349a3ff9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b164cbbc00f4e8eb3254ddd349a3ff9, 8b8b3ad3aa8f4053b2167dd550f2086f; no voters:
I20240501 07:59:04.896250 8520 raft_consensus.cc:2781] T 6e2ac8c8857d45b88d3b51147d2609aa P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:04.896308 8520 raft_consensus.cc:482] T 6e2ac8c8857d45b88d3b51147d2609aa P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:04.896356 8520 raft_consensus.cc:3037] T 6e2ac8c8857d45b88d3b51147d2609aa P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:04.897338 8520 raft_consensus.cc:504] T 6e2ac8c8857d45b88d3b51147d2609aa P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } }
I20240501 07:59:04.897680 8520 leader_election.cc:288] T 6e2ac8c8857d45b88d3b51147d2609aa P 3b164cbbc00f4e8eb3254ddd349a3ff9 [CANDIDATE]: Term 1 election: Requested vote from peers 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129:46875), 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130:32941)
I20240501 07:59:04.897982 6313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e2ac8c8857d45b88d3b51147d2609aa" candidate_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f"
I20240501 07:59:04.898092 6313 raft_consensus.cc:3037] T 6e2ac8c8857d45b88d3b51147d2609aa P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:04.898154 6444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e2ac8c8857d45b88d3b51147d2609aa" candidate_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8315d22c56f7439a9e1fc82f153f018c"
I20240501 07:59:04.898275 6444 raft_consensus.cc:3037] T 6e2ac8c8857d45b88d3b51147d2609aa P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:04.898947 6313 raft_consensus.cc:2454] T 6e2ac8c8857d45b88d3b51147d2609aa P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b164cbbc00f4e8eb3254ddd349a3ff9 in term 1.
I20240501 07:59:04.899107 6444 raft_consensus.cc:2454] T 6e2ac8c8857d45b88d3b51147d2609aa P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b164cbbc00f4e8eb3254ddd349a3ff9 in term 1.
I20240501 07:59:04.899228 6510 leader_election.cc:302] T 6e2ac8c8857d45b88d3b51147d2609aa P 3b164cbbc00f4e8eb3254ddd349a3ff9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b164cbbc00f4e8eb3254ddd349a3ff9, 8b8b3ad3aa8f4053b2167dd550f2086f; no voters:
I20240501 07:59:04.899482 8520 raft_consensus.cc:2781] T 6e2ac8c8857d45b88d3b51147d2609aa P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:04.899587 8520 raft_consensus.cc:686] T 6e2ac8c8857d45b88d3b51147d2609aa P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 LEADER]: Becoming Leader. State: Replica: 3b164cbbc00f4e8eb3254ddd349a3ff9, State: Running, Role: LEADER
I20240501 07:59:04.899724 8520 consensus_queue.cc:237] T 6e2ac8c8857d45b88d3b51147d2609aa P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } }
I20240501 07:59:04.902019 6170 catalog_manager.cc:5484] T 6e2ac8c8857d45b88d3b51147d2609aa P 3b164cbbc00f4e8eb3254ddd349a3ff9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } health_report { overall_health: HEALTHY } } }
I20240501 07:59:04.909597 8520 raft_consensus.cc:482] T dc550f54e8024d47b1955905e7858a1e P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:04.909737 8520 raft_consensus.cc:504] T dc550f54e8024d47b1955905e7858a1e P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:59:04.910179 8520 leader_election.cc:288] T dc550f54e8024d47b1955905e7858a1e P 3b164cbbc00f4e8eb3254ddd349a3ff9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130:32941), 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132:36773)
I20240501 07:59:04.910668 6708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc550f54e8024d47b1955905e7858a1e" candidate_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f9cb72be98941b5a8e504cbe34132b9" is_pre_election: true
I20240501 07:59:04.910671 6444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc550f54e8024d47b1955905e7858a1e" candidate_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8315d22c56f7439a9e1fc82f153f018c" is_pre_election: true
I20240501 07:59:04.910835 6444 raft_consensus.cc:2454] T dc550f54e8024d47b1955905e7858a1e P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b164cbbc00f4e8eb3254ddd349a3ff9 in term 0.
I20240501 07:59:04.910833 6708 raft_consensus.cc:2454] T dc550f54e8024d47b1955905e7858a1e P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b164cbbc00f4e8eb3254ddd349a3ff9 in term 0.
I20240501 07:59:04.911291 6512 leader_election.cc:302] T dc550f54e8024d47b1955905e7858a1e P 3b164cbbc00f4e8eb3254ddd349a3ff9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b164cbbc00f4e8eb3254ddd349a3ff9, 8315d22c56f7439a9e1fc82f153f018c; no voters:
I20240501 07:59:04.911484 8520 raft_consensus.cc:2781] T dc550f54e8024d47b1955905e7858a1e P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:04.911543 8520 raft_consensus.cc:482] T dc550f54e8024d47b1955905e7858a1e P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:04.911598 8520 raft_consensus.cc:3037] T dc550f54e8024d47b1955905e7858a1e P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:04.912410 8520 raft_consensus.cc:504] T dc550f54e8024d47b1955905e7858a1e P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:59:04.912734 8520 leader_election.cc:288] T dc550f54e8024d47b1955905e7858a1e P 3b164cbbc00f4e8eb3254ddd349a3ff9 [CANDIDATE]: Term 1 election: Requested vote from peers 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130:32941), 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132:36773)
I20240501 07:59:04.913210 6708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc550f54e8024d47b1955905e7858a1e" candidate_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f9cb72be98941b5a8e504cbe34132b9"
I20240501 07:59:04.913347 6708 raft_consensus.cc:3037] T dc550f54e8024d47b1955905e7858a1e P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:04.914249 6708 raft_consensus.cc:2454] T dc550f54e8024d47b1955905e7858a1e P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b164cbbc00f4e8eb3254ddd349a3ff9 in term 1.
I20240501 07:59:04.914526 6444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc550f54e8024d47b1955905e7858a1e" candidate_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8315d22c56f7439a9e1fc82f153f018c"
I20240501 07:59:04.914556 6512 leader_election.cc:302] T dc550f54e8024d47b1955905e7858a1e P 3b164cbbc00f4e8eb3254ddd349a3ff9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f9cb72be98941b5a8e504cbe34132b9, 3b164cbbc00f4e8eb3254ddd349a3ff9; no voters:
I20240501 07:59:04.914652 6444 raft_consensus.cc:3037] T dc550f54e8024d47b1955905e7858a1e P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:04.914713 8520 raft_consensus.cc:2781] T dc550f54e8024d47b1955905e7858a1e P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:04.914835 8520 raft_consensus.cc:686] T dc550f54e8024d47b1955905e7858a1e P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 LEADER]: Becoming Leader. State: Replica: 3b164cbbc00f4e8eb3254ddd349a3ff9, State: Running, Role: LEADER
I20240501 07:59:04.914981 8520 consensus_queue.cc:237] T dc550f54e8024d47b1955905e7858a1e P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:59:04.915596 6444 raft_consensus.cc:2454] T dc550f54e8024d47b1955905e7858a1e P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b164cbbc00f4e8eb3254ddd349a3ff9 in term 1.
I20240501 07:59:04.917555 6170 catalog_manager.cc:5484] T dc550f54e8024d47b1955905e7858a1e P 3b164cbbc00f4e8eb3254ddd349a3ff9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } health_report { overall_health: UNKNOWN } } }
I20240501 07:59:05.000519 8444 raft_consensus.cc:482] T 7dd6566ccb6e43388146a6dfd1c4a88c P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:05.000666 8444 raft_consensus.cc:504] T 7dd6566ccb6e43388146a6dfd1c4a88c P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } }
I20240501 07:59:05.001111 8444 leader_election.cc:288] T 7dd6566ccb6e43388146a6dfd1c4a88c P 1f9cb72be98941b5a8e504cbe34132b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130:32941), 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131:34471)
I20240501 07:59:05.001524 6577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dd6566ccb6e43388146a6dfd1c4a88c" candidate_uuid: "1f9cb72be98941b5a8e504cbe34132b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" is_pre_election: true
I20240501 07:59:05.001538 6444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dd6566ccb6e43388146a6dfd1c4a88c" candidate_uuid: "1f9cb72be98941b5a8e504cbe34132b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8315d22c56f7439a9e1fc82f153f018c" is_pre_election: true
I20240501 07:59:05.001688 6444 raft_consensus.cc:2454] T 7dd6566ccb6e43388146a6dfd1c4a88c P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f9cb72be98941b5a8e504cbe34132b9 in term 0.
I20240501 07:59:05.001693 6577 raft_consensus.cc:2454] T 7dd6566ccb6e43388146a6dfd1c4a88c P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f9cb72be98941b5a8e504cbe34132b9 in term 0.
I20240501 07:59:05.001974 6643 leader_election.cc:302] T 7dd6566ccb6e43388146a6dfd1c4a88c P 1f9cb72be98941b5a8e504cbe34132b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f9cb72be98941b5a8e504cbe34132b9, 8315d22c56f7439a9e1fc82f153f018c; no voters:
I20240501 07:59:05.002151 8444 raft_consensus.cc:2781] T 7dd6566ccb6e43388146a6dfd1c4a88c P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:05.002209 8444 raft_consensus.cc:482] T 7dd6566ccb6e43388146a6dfd1c4a88c P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:05.002235 8444 raft_consensus.cc:3037] T 7dd6566ccb6e43388146a6dfd1c4a88c P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:05.003145 8444 raft_consensus.cc:504] T 7dd6566ccb6e43388146a6dfd1c4a88c P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } }
I20240501 07:59:05.003499 8444 leader_election.cc:288] T 7dd6566ccb6e43388146a6dfd1c4a88c P 1f9cb72be98941b5a8e504cbe34132b9 [CANDIDATE]: Term 1 election: Requested vote from peers 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130:32941), 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131:34471)
I20240501 07:59:05.003775 6445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dd6566ccb6e43388146a6dfd1c4a88c" candidate_uuid: "1f9cb72be98941b5a8e504cbe34132b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8315d22c56f7439a9e1fc82f153f018c"
I20240501 07:59:05.003815 6577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dd6566ccb6e43388146a6dfd1c4a88c" candidate_uuid: "1f9cb72be98941b5a8e504cbe34132b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9"
I20240501 07:59:05.003906 6445 raft_consensus.cc:3037] T 7dd6566ccb6e43388146a6dfd1c4a88c P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:05.003926 6577 raft_consensus.cc:3037] T 7dd6566ccb6e43388146a6dfd1c4a88c P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:05.004834 6445 raft_consensus.cc:2454] T 7dd6566ccb6e43388146a6dfd1c4a88c P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f9cb72be98941b5a8e504cbe34132b9 in term 1.
I20240501 07:59:05.004846 6577 raft_consensus.cc:2454] T 7dd6566ccb6e43388146a6dfd1c4a88c P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f9cb72be98941b5a8e504cbe34132b9 in term 1.
I20240501 07:59:05.005208 6643 leader_election.cc:302] T 7dd6566ccb6e43388146a6dfd1c4a88c P 1f9cb72be98941b5a8e504cbe34132b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f9cb72be98941b5a8e504cbe34132b9, 8315d22c56f7439a9e1fc82f153f018c; no voters:
I20240501 07:59:05.005415 8444 raft_consensus.cc:2781] T 7dd6566ccb6e43388146a6dfd1c4a88c P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:05.005514 8444 raft_consensus.cc:686] T 7dd6566ccb6e43388146a6dfd1c4a88c P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 LEADER]: Becoming Leader. State: Replica: 1f9cb72be98941b5a8e504cbe34132b9, State: Running, Role: LEADER
I20240501 07:59:05.005662 8444 consensus_queue.cc:237] T 7dd6566ccb6e43388146a6dfd1c4a88c P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } }
I20240501 07:59:05.007665 6170 catalog_manager.cc:5484] T 7dd6566ccb6e43388146a6dfd1c4a88c P 1f9cb72be98941b5a8e504cbe34132b9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132). New cstate: current_term: 1 leader_uuid: "1f9cb72be98941b5a8e504cbe34132b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } health_report { overall_health: UNKNOWN } } }
W20240501 07:59:05.139458 8549 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:59:05.139758 8549 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:59:05.143110 8549 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_17'
I20240501 07:59:05.240789 6445 raft_consensus.cc:1261] T 6e2ac8c8857d45b88d3b51147d2609aa P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Refusing update from remote peer 3b164cbbc00f4e8eb3254ddd349a3ff9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:05.241014 6708 raft_consensus.cc:1261] T dc550f54e8024d47b1955905e7858a1e P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Refusing update from remote peer 3b164cbbc00f4e8eb3254ddd349a3ff9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:05.241544 6445 raft_consensus.cc:1261] T dc550f54e8024d47b1955905e7858a1e P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Refusing update from remote peer 3b164cbbc00f4e8eb3254ddd349a3ff9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:05.241510 8508 consensus_queue.cc:1035] T 6e2ac8c8857d45b88d3b51147d2609aa P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:05.241847 8508 consensus_queue.cc:1035] T dc550f54e8024d47b1955905e7858a1e P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:05.246170 6577 raft_consensus.cc:1261] T 7dd6566ccb6e43388146a6dfd1c4a88c P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Refusing update from remote peer 1f9cb72be98941b5a8e504cbe34132b9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:05.246560 8444 consensus_queue.cc:1035] T 7dd6566ccb6e43388146a6dfd1c4a88c P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:05.249161 6444 raft_consensus.cc:1261] T 7dd6566ccb6e43388146a6dfd1c4a88c P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Refusing update from remote peer 1f9cb72be98941b5a8e504cbe34132b9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:05.249716 8444 consensus_queue.cc:1035] T 7dd6566ccb6e43388146a6dfd1c4a88c P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:05.251231 6314 raft_consensus.cc:1261] T 6e2ac8c8857d45b88d3b51147d2609aa P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Refusing update from remote peer 3b164cbbc00f4e8eb3254ddd349a3ff9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:05.251762 8166 consensus_queue.cc:1035] T 6e2ac8c8857d45b88d3b51147d2609aa P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:05.259743 8468 consensus_queue.cc:1035] T dc550f54e8024d47b1955905e7858a1e P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:05.264516 8509 mvcc.cc:205] Tried to move back new op lower bound from 7022798214103273472 to 7022798212773220352. Current Snapshot: MvccSnapshot[applied={T|T < 7022798214103273472}]
INSERT report
rows total: 2000
time total: 129.829 ms
time per row: 0.0649144 ms
I20240501 07:59:05.318017 6167 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55950:
name: "rebalance_test_table_18"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 07:59:05.323695 6293 tablet_service.cc:1467] Processing CreateTablet for tablet ae46990350284295bf14aa64d4b6c93e (DEFAULT_TABLE table=rebalance_test_table_18 [id=25ecaaa6eafe48ecb3f3ff353a6d1170]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:05.323669 6687 tablet_service.cc:1467] Processing CreateTablet for tablet ae46990350284295bf14aa64d4b6c93e (DEFAULT_TABLE table=rebalance_test_table_18 [id=25ecaaa6eafe48ecb3f3ff353a6d1170]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:05.323998 6687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae46990350284295bf14aa64d4b6c93e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:05.323998 6293 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae46990350284295bf14aa64d4b6c93e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:05.324200 6295 tablet_service.cc:1467] Processing CreateTablet for tablet aa1b8716228b4fd9af95870b7a613fb2 (DEFAULT_TABLE table=rebalance_test_table_18 [id=25ecaaa6eafe48ecb3f3ff353a6d1170]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:05.324451 6295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa1b8716228b4fd9af95870b7a613fb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:05.324200 6294 tablet_service.cc:1467] Processing CreateTablet for tablet b284ed2cb0264ced99a4ffd90983d902 (DEFAULT_TABLE table=rebalance_test_table_18 [id=25ecaaa6eafe48ecb3f3ff353a6d1170]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:05.325872 6294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b284ed2cb0264ced99a4ffd90983d902. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:05.326246 8535 tablet_bootstrap.cc:492] T ae46990350284295bf14aa64d4b6c93e P 8b8b3ad3aa8f4053b2167dd550f2086f: Bootstrap starting.
I20240501 07:59:05.326678 6556 tablet_service.cc:1467] Processing CreateTablet for tablet b284ed2cb0264ced99a4ffd90983d902 (DEFAULT_TABLE table=rebalance_test_table_18 [id=25ecaaa6eafe48ecb3f3ff353a6d1170]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:05.326983 6556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b284ed2cb0264ced99a4ffd90983d902. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:05.327036 8535 tablet_bootstrap.cc:654] T ae46990350284295bf14aa64d4b6c93e P 8b8b3ad3aa8f4053b2167dd550f2086f: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:05.327483 8534 tablet_bootstrap.cc:492] T ae46990350284295bf14aa64d4b6c93e P 1f9cb72be98941b5a8e504cbe34132b9: Bootstrap starting.
I20240501 07:59:05.328199 8534 tablet_bootstrap.cc:654] T ae46990350284295bf14aa64d4b6c93e P 1f9cb72be98941b5a8e504cbe34132b9: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:05.328330 6425 tablet_service.cc:1467] Processing CreateTablet for tablet ae46990350284295bf14aa64d4b6c93e (DEFAULT_TABLE table=rebalance_test_table_18 [id=25ecaaa6eafe48ecb3f3ff353a6d1170]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:05.328361 6426 tablet_service.cc:1467] Processing CreateTablet for tablet aa1b8716228b4fd9af95870b7a613fb2 (DEFAULT_TABLE table=rebalance_test_table_18 [id=25ecaaa6eafe48ecb3f3ff353a6d1170]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:05.328670 6425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae46990350284295bf14aa64d4b6c93e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:05.329355 6686 tablet_service.cc:1467] Processing CreateTablet for tablet aa1b8716228b4fd9af95870b7a613fb2 (DEFAULT_TABLE table=rebalance_test_table_18 [id=25ecaaa6eafe48ecb3f3ff353a6d1170]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:05.329445 8535 tablet_bootstrap.cc:492] T ae46990350284295bf14aa64d4b6c93e P 8b8b3ad3aa8f4053b2167dd550f2086f: No bootstrap required, opened a new log
I20240501 07:59:05.329528 8535 ts_tablet_manager.cc:1367] T ae46990350284295bf14aa64d4b6c93e P 8b8b3ad3aa8f4053b2167dd550f2086f: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240501 07:59:05.329589 6686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa1b8716228b4fd9af95870b7a613fb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:05.329943 8535 raft_consensus.cc:348] T ae46990350284295bf14aa64d4b6c93e P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } }
I20240501 07:59:05.330060 8535 raft_consensus.cc:374] T ae46990350284295bf14aa64d4b6c93e P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:05.330090 8535 raft_consensus.cc:729] T ae46990350284295bf14aa64d4b6c93e P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b8b3ad3aa8f4053b2167dd550f2086f, State: Initialized, Role: FOLLOWER
I20240501 07:59:05.330163 6426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa1b8716228b4fd9af95870b7a613fb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:05.330194 8535 consensus_queue.cc:260] T ae46990350284295bf14aa64d4b6c93e P 8b8b3ad3aa8f4053b2167dd550f2086f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } }
I20240501 07:59:05.330466 8535 ts_tablet_manager.cc:1398] T ae46990350284295bf14aa64d4b6c93e P 8b8b3ad3aa8f4053b2167dd550f2086f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:59:05.330646 8535 tablet_bootstrap.cc:492] T aa1b8716228b4fd9af95870b7a613fb2 P 8b8b3ad3aa8f4053b2167dd550f2086f: Bootstrap starting.
I20240501 07:59:05.331010 6687 tablet_service.cc:1467] Processing CreateTablet for tablet b284ed2cb0264ced99a4ffd90983d902 (DEFAULT_TABLE table=rebalance_test_table_18 [id=25ecaaa6eafe48ecb3f3ff353a6d1170]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:05.331249 6687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b284ed2cb0264ced99a4ffd90983d902. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:05.331734 8536 tablet_bootstrap.cc:492] T b284ed2cb0264ced99a4ffd90983d902 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Bootstrap starting.
I20240501 07:59:05.331837 8533 tablet_bootstrap.cc:492] T ae46990350284295bf14aa64d4b6c93e P 8315d22c56f7439a9e1fc82f153f018c: Bootstrap starting.
I20240501 07:59:05.332599 8536 tablet_bootstrap.cc:654] T b284ed2cb0264ced99a4ffd90983d902 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:05.332755 8533 tablet_bootstrap.cc:654] T ae46990350284295bf14aa64d4b6c93e P 8315d22c56f7439a9e1fc82f153f018c: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:05.333376 8535 tablet_bootstrap.cc:654] T aa1b8716228b4fd9af95870b7a613fb2 P 8b8b3ad3aa8f4053b2167dd550f2086f: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:05.334318 8533 tablet_bootstrap.cc:492] T ae46990350284295bf14aa64d4b6c93e P 8315d22c56f7439a9e1fc82f153f018c: No bootstrap required, opened a new log
I20240501 07:59:05.334403 8533 ts_tablet_manager.cc:1367] T ae46990350284295bf14aa64d4b6c93e P 8315d22c56f7439a9e1fc82f153f018c: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 07:59:05.334489 8534 tablet_bootstrap.cc:492] T ae46990350284295bf14aa64d4b6c93e P 1f9cb72be98941b5a8e504cbe34132b9: No bootstrap required, opened a new log
I20240501 07:59:05.334558 8534 ts_tablet_manager.cc:1367] T ae46990350284295bf14aa64d4b6c93e P 1f9cb72be98941b5a8e504cbe34132b9: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20240501 07:59:05.334841 8533 raft_consensus.cc:348] T ae46990350284295bf14aa64d4b6c93e P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } }
I20240501 07:59:05.334965 8533 raft_consensus.cc:374] T ae46990350284295bf14aa64d4b6c93e P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:05.335013 8533 raft_consensus.cc:729] T ae46990350284295bf14aa64d4b6c93e P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8315d22c56f7439a9e1fc82f153f018c, State: Initialized, Role: FOLLOWER
I20240501 07:59:05.334969 8534 raft_consensus.cc:348] T ae46990350284295bf14aa64d4b6c93e P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } }
I20240501 07:59:05.335072 8534 raft_consensus.cc:374] T ae46990350284295bf14aa64d4b6c93e P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:05.335107 8534 raft_consensus.cc:729] T ae46990350284295bf14aa64d4b6c93e P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f9cb72be98941b5a8e504cbe34132b9, State: Initialized, Role: FOLLOWER
I20240501 07:59:05.335157 8533 consensus_queue.cc:260] T ae46990350284295bf14aa64d4b6c93e P 8315d22c56f7439a9e1fc82f153f018c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } }
I20240501 07:59:05.335237 8534 consensus_queue.cc:260] T ae46990350284295bf14aa64d4b6c93e P 1f9cb72be98941b5a8e504cbe34132b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } }
I20240501 07:59:05.335477 8533 ts_tablet_manager.cc:1398] T ae46990350284295bf14aa64d4b6c93e P 8315d22c56f7439a9e1fc82f153f018c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:05.335520 8534 ts_tablet_manager.cc:1398] T ae46990350284295bf14aa64d4b6c93e P 1f9cb72be98941b5a8e504cbe34132b9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:05.335806 8534 tablet_bootstrap.cc:492] T aa1b8716228b4fd9af95870b7a613fb2 P 1f9cb72be98941b5a8e504cbe34132b9: Bootstrap starting.
I20240501 07:59:05.336073 8533 tablet_bootstrap.cc:492] T aa1b8716228b4fd9af95870b7a613fb2 P 8315d22c56f7439a9e1fc82f153f018c: Bootstrap starting.
I20240501 07:59:05.336863 8534 tablet_bootstrap.cc:654] T aa1b8716228b4fd9af95870b7a613fb2 P 1f9cb72be98941b5a8e504cbe34132b9: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:05.337990 8535 tablet_bootstrap.cc:492] T aa1b8716228b4fd9af95870b7a613fb2 P 8b8b3ad3aa8f4053b2167dd550f2086f: No bootstrap required, opened a new log
I20240501 07:59:05.337997 8533 tablet_bootstrap.cc:654] T aa1b8716228b4fd9af95870b7a613fb2 P 8315d22c56f7439a9e1fc82f153f018c: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:05.338064 8535 ts_tablet_manager.cc:1367] T aa1b8716228b4fd9af95870b7a613fb2 P 8b8b3ad3aa8f4053b2167dd550f2086f: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240501 07:59:05.338512 8535 raft_consensus.cc:348] T aa1b8716228b4fd9af95870b7a613fb2 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } }
I20240501 07:59:05.338620 8535 raft_consensus.cc:374] T aa1b8716228b4fd9af95870b7a613fb2 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:05.338675 8535 raft_consensus.cc:729] T aa1b8716228b4fd9af95870b7a613fb2 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b8b3ad3aa8f4053b2167dd550f2086f, State: Initialized, Role: FOLLOWER
I20240501 07:59:05.338794 8535 consensus_queue.cc:260] T aa1b8716228b4fd9af95870b7a613fb2 P 8b8b3ad3aa8f4053b2167dd550f2086f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } }
I20240501 07:59:05.339073 8535 ts_tablet_manager.cc:1398] T aa1b8716228b4fd9af95870b7a613fb2 P 8b8b3ad3aa8f4053b2167dd550f2086f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:05.339260 8535 tablet_bootstrap.cc:492] T b284ed2cb0264ced99a4ffd90983d902 P 8b8b3ad3aa8f4053b2167dd550f2086f: Bootstrap starting.
I20240501 07:59:05.340186 8535 tablet_bootstrap.cc:654] T b284ed2cb0264ced99a4ffd90983d902 P 8b8b3ad3aa8f4053b2167dd550f2086f: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:05.340344 8536 tablet_bootstrap.cc:492] T b284ed2cb0264ced99a4ffd90983d902 P 3b164cbbc00f4e8eb3254ddd349a3ff9: No bootstrap required, opened a new log
I20240501 07:59:05.340435 8536 ts_tablet_manager.cc:1367] T b284ed2cb0264ced99a4ffd90983d902 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240501 07:59:05.340924 8536 raft_consensus.cc:348] T b284ed2cb0264ced99a4ffd90983d902 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:59:05.341055 8536 raft_consensus.cc:374] T b284ed2cb0264ced99a4ffd90983d902 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:05.341107 8536 raft_consensus.cc:729] T b284ed2cb0264ced99a4ffd90983d902 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b164cbbc00f4e8eb3254ddd349a3ff9, State: Initialized, Role: FOLLOWER
I20240501 07:59:05.341231 8535 tablet_bootstrap.cc:492] T b284ed2cb0264ced99a4ffd90983d902 P 8b8b3ad3aa8f4053b2167dd550f2086f: No bootstrap required, opened a new log
I20240501 07:59:05.341220 8536 consensus_queue.cc:260] T b284ed2cb0264ced99a4ffd90983d902 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:59:05.341300 8535 ts_tablet_manager.cc:1367] T b284ed2cb0264ced99a4ffd90983d902 P 8b8b3ad3aa8f4053b2167dd550f2086f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:59:05.341327 8534 tablet_bootstrap.cc:492] T aa1b8716228b4fd9af95870b7a613fb2 P 1f9cb72be98941b5a8e504cbe34132b9: No bootstrap required, opened a new log
I20240501 07:59:05.341435 8534 ts_tablet_manager.cc:1367] T aa1b8716228b4fd9af95870b7a613fb2 P 1f9cb72be98941b5a8e504cbe34132b9: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 07:59:05.341512 8536 ts_tablet_manager.cc:1398] T b284ed2cb0264ced99a4ffd90983d902 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:05.341704 8535 raft_consensus.cc:348] T b284ed2cb0264ced99a4ffd90983d902 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:59:05.341794 8535 raft_consensus.cc:374] T b284ed2cb0264ced99a4ffd90983d902 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:05.341827 8535 raft_consensus.cc:729] T b284ed2cb0264ced99a4ffd90983d902 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b8b3ad3aa8f4053b2167dd550f2086f, State: Initialized, Role: FOLLOWER
I20240501 07:59:05.341840 8534 raft_consensus.cc:348] T aa1b8716228b4fd9af95870b7a613fb2 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } }
I20240501 07:59:05.341931 8534 raft_consensus.cc:374] T aa1b8716228b4fd9af95870b7a613fb2 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:05.341961 8534 raft_consensus.cc:729] T aa1b8716228b4fd9af95870b7a613fb2 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f9cb72be98941b5a8e504cbe34132b9, State: Initialized, Role: FOLLOWER
I20240501 07:59:05.341954 8535 consensus_queue.cc:260] T b284ed2cb0264ced99a4ffd90983d902 P 8b8b3ad3aa8f4053b2167dd550f2086f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:59:05.342065 8534 consensus_queue.cc:260] T aa1b8716228b4fd9af95870b7a613fb2 P 1f9cb72be98941b5a8e504cbe34132b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } }
I20240501 07:59:05.342238 8535 ts_tablet_manager.cc:1398] T b284ed2cb0264ced99a4ffd90983d902 P 8b8b3ad3aa8f4053b2167dd550f2086f: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 07:59:05.342327 8534 ts_tablet_manager.cc:1398] T aa1b8716228b4fd9af95870b7a613fb2 P 1f9cb72be98941b5a8e504cbe34132b9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:05.342468 8534 tablet_bootstrap.cc:492] T b284ed2cb0264ced99a4ffd90983d902 P 1f9cb72be98941b5a8e504cbe34132b9: Bootstrap starting.
I20240501 07:59:05.343379 8534 tablet_bootstrap.cc:654] T b284ed2cb0264ced99a4ffd90983d902 P 1f9cb72be98941b5a8e504cbe34132b9: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:05.344280 8533 tablet_bootstrap.cc:492] T aa1b8716228b4fd9af95870b7a613fb2 P 8315d22c56f7439a9e1fc82f153f018c: No bootstrap required, opened a new log
I20240501 07:59:05.344368 8533 ts_tablet_manager.cc:1367] T aa1b8716228b4fd9af95870b7a613fb2 P 8315d22c56f7439a9e1fc82f153f018c: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.002s
I20240501 07:59:05.344797 8533 raft_consensus.cc:348] T aa1b8716228b4fd9af95870b7a613fb2 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } }
I20240501 07:59:05.344945 8533 raft_consensus.cc:374] T aa1b8716228b4fd9af95870b7a613fb2 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:05.345855 8533 raft_consensus.cc:729] T aa1b8716228b4fd9af95870b7a613fb2 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8315d22c56f7439a9e1fc82f153f018c, State: Initialized, Role: FOLLOWER
I20240501 07:59:05.346022 8533 consensus_queue.cc:260] T aa1b8716228b4fd9af95870b7a613fb2 P 8315d22c56f7439a9e1fc82f153f018c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } }
I20240501 07:59:05.346343 8533 ts_tablet_manager.cc:1398] T aa1b8716228b4fd9af95870b7a613fb2 P 8315d22c56f7439a9e1fc82f153f018c: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 07:59:05.347262 8534 tablet_bootstrap.cc:492] T b284ed2cb0264ced99a4ffd90983d902 P 1f9cb72be98941b5a8e504cbe34132b9: No bootstrap required, opened a new log
I20240501 07:59:05.347373 8534 ts_tablet_manager.cc:1367] T b284ed2cb0264ced99a4ffd90983d902 P 1f9cb72be98941b5a8e504cbe34132b9: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240501 07:59:05.347785 8534 raft_consensus.cc:348] T b284ed2cb0264ced99a4ffd90983d902 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:59:05.347887 8534 raft_consensus.cc:374] T b284ed2cb0264ced99a4ffd90983d902 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:05.347921 8534 raft_consensus.cc:729] T b284ed2cb0264ced99a4ffd90983d902 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f9cb72be98941b5a8e504cbe34132b9, State: Initialized, Role: FOLLOWER
I20240501 07:59:05.348052 8534 consensus_queue.cc:260] T b284ed2cb0264ced99a4ffd90983d902 P 1f9cb72be98941b5a8e504cbe34132b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:59:05.348383 8534 ts_tablet_manager.cc:1398] T b284ed2cb0264ced99a4ffd90983d902 P 1f9cb72be98941b5a8e504cbe34132b9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:05.385514 8385 raft_consensus.cc:482] T ae46990350284295bf14aa64d4b6c93e P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:05.385660 8385 raft_consensus.cc:504] T ae46990350284295bf14aa64d4b6c93e P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } }
I20240501 07:59:05.386080 8385 leader_election.cc:288] T ae46990350284295bf14aa64d4b6c93e P 1f9cb72be98941b5a8e504cbe34132b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129:46875), 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130:32941)
I20240501 07:59:05.386535 6314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae46990350284295bf14aa64d4b6c93e" candidate_uuid: "1f9cb72be98941b5a8e504cbe34132b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" is_pre_election: true
I20240501 07:59:05.386535 6444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae46990350284295bf14aa64d4b6c93e" candidate_uuid: "1f9cb72be98941b5a8e504cbe34132b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8315d22c56f7439a9e1fc82f153f018c" is_pre_election: true
I20240501 07:59:05.386714 6444 raft_consensus.cc:2454] T ae46990350284295bf14aa64d4b6c93e P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f9cb72be98941b5a8e504cbe34132b9 in term 0.
I20240501 07:59:05.386714 6314 raft_consensus.cc:2454] T ae46990350284295bf14aa64d4b6c93e P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f9cb72be98941b5a8e504cbe34132b9 in term 0.
I20240501 07:59:05.387032 6643 leader_election.cc:302] T ae46990350284295bf14aa64d4b6c93e P 1f9cb72be98941b5a8e504cbe34132b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f9cb72be98941b5a8e504cbe34132b9, 8315d22c56f7439a9e1fc82f153f018c; no voters:
I20240501 07:59:05.387223 8444 raft_consensus.cc:2781] T ae46990350284295bf14aa64d4b6c93e P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:05.387300 8444 raft_consensus.cc:482] T ae46990350284295bf14aa64d4b6c93e P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:05.387343 8444 raft_consensus.cc:3037] T ae46990350284295bf14aa64d4b6c93e P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:05.388337 8444 raft_consensus.cc:504] T ae46990350284295bf14aa64d4b6c93e P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } }
I20240501 07:59:05.388698 8444 leader_election.cc:288] T ae46990350284295bf14aa64d4b6c93e P 1f9cb72be98941b5a8e504cbe34132b9 [CANDIDATE]: Term 1 election: Requested vote from peers 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129:46875), 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130:32941)
I20240501 07:59:05.389031 6314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae46990350284295bf14aa64d4b6c93e" candidate_uuid: "1f9cb72be98941b5a8e504cbe34132b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f"
I20240501 07:59:05.389150 6314 raft_consensus.cc:3037] T ae46990350284295bf14aa64d4b6c93e P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:05.389137 6444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae46990350284295bf14aa64d4b6c93e" candidate_uuid: "1f9cb72be98941b5a8e504cbe34132b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8315d22c56f7439a9e1fc82f153f018c"
I20240501 07:59:05.389276 6444 raft_consensus.cc:3037] T ae46990350284295bf14aa64d4b6c93e P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:05.389994 6314 raft_consensus.cc:2454] T ae46990350284295bf14aa64d4b6c93e P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f9cb72be98941b5a8e504cbe34132b9 in term 1.
I20240501 07:59:05.390074 6444 raft_consensus.cc:2454] T ae46990350284295bf14aa64d4b6c93e P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f9cb72be98941b5a8e504cbe34132b9 in term 1.
I20240501 07:59:05.390329 6643 leader_election.cc:302] T ae46990350284295bf14aa64d4b6c93e P 1f9cb72be98941b5a8e504cbe34132b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f9cb72be98941b5a8e504cbe34132b9, 8315d22c56f7439a9e1fc82f153f018c; no voters:
I20240501 07:59:05.390529 8444 raft_consensus.cc:2781] T ae46990350284295bf14aa64d4b6c93e P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:05.390609 8444 raft_consensus.cc:686] T ae46990350284295bf14aa64d4b6c93e P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 LEADER]: Becoming Leader. State: Replica: 1f9cb72be98941b5a8e504cbe34132b9, State: Running, Role: LEADER
I20240501 07:59:05.390749 8444 consensus_queue.cc:237] T ae46990350284295bf14aa64d4b6c93e P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } }
I20240501 07:59:05.392812 6166 catalog_manager.cc:5484] T ae46990350284295bf14aa64d4b6c93e P 1f9cb72be98941b5a8e504cbe34132b9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132). New cstate: current_term: 1 leader_uuid: "1f9cb72be98941b5a8e504cbe34132b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } health_report { overall_health: UNKNOWN } } }
I20240501 07:59:05.454428 8385 raft_consensus.cc:482] T b284ed2cb0264ced99a4ffd90983d902 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:05.454561 8385 raft_consensus.cc:504] T b284ed2cb0264ced99a4ffd90983d902 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:59:05.454968 8385 leader_election.cc:288] T b284ed2cb0264ced99a4ffd90983d902 P 1f9cb72be98941b5a8e504cbe34132b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129:46875), 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131:34471)
I20240501 07:59:05.455333 6314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b284ed2cb0264ced99a4ffd90983d902" candidate_uuid: "1f9cb72be98941b5a8e504cbe34132b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" is_pre_election: true
I20240501 07:59:05.455367 6577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b284ed2cb0264ced99a4ffd90983d902" candidate_uuid: "1f9cb72be98941b5a8e504cbe34132b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" is_pre_election: true
I20240501 07:59:05.455485 6314 raft_consensus.cc:2454] T b284ed2cb0264ced99a4ffd90983d902 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f9cb72be98941b5a8e504cbe34132b9 in term 0.
I20240501 07:59:05.455500 6577 raft_consensus.cc:2454] T b284ed2cb0264ced99a4ffd90983d902 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f9cb72be98941b5a8e504cbe34132b9 in term 0.
I20240501 07:59:05.455773 6641 leader_election.cc:302] T b284ed2cb0264ced99a4ffd90983d902 P 1f9cb72be98941b5a8e504cbe34132b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f9cb72be98941b5a8e504cbe34132b9, 8b8b3ad3aa8f4053b2167dd550f2086f; no voters:
I20240501 07:59:05.455950 8385 raft_consensus.cc:2781] T b284ed2cb0264ced99a4ffd90983d902 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:05.456017 8385 raft_consensus.cc:482] T b284ed2cb0264ced99a4ffd90983d902 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:05.456068 8385 raft_consensus.cc:3037] T b284ed2cb0264ced99a4ffd90983d902 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:05.456939 8385 raft_consensus.cc:504] T b284ed2cb0264ced99a4ffd90983d902 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:59:05.457298 8385 leader_election.cc:288] T b284ed2cb0264ced99a4ffd90983d902 P 1f9cb72be98941b5a8e504cbe34132b9 [CANDIDATE]: Term 1 election: Requested vote from peers 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129:46875), 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131:34471)
I20240501 07:59:05.457646 6577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b284ed2cb0264ced99a4ffd90983d902" candidate_uuid: "1f9cb72be98941b5a8e504cbe34132b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9"
I20240501 07:59:05.457767 6577 raft_consensus.cc:3037] T b284ed2cb0264ced99a4ffd90983d902 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:05.457648 6314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b284ed2cb0264ced99a4ffd90983d902" candidate_uuid: "1f9cb72be98941b5a8e504cbe34132b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f"
I20240501 07:59:05.457893 6314 raft_consensus.cc:3037] T b284ed2cb0264ced99a4ffd90983d902 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:05.458791 6314 raft_consensus.cc:2454] T b284ed2cb0264ced99a4ffd90983d902 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f9cb72be98941b5a8e504cbe34132b9 in term 1.
I20240501 07:59:05.458791 6577 raft_consensus.cc:2454] T b284ed2cb0264ced99a4ffd90983d902 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f9cb72be98941b5a8e504cbe34132b9 in term 1.
I20240501 07:59:05.459173 6641 leader_election.cc:302] T b284ed2cb0264ced99a4ffd90983d902 P 1f9cb72be98941b5a8e504cbe34132b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f9cb72be98941b5a8e504cbe34132b9, 8b8b3ad3aa8f4053b2167dd550f2086f; no voters:
I20240501 07:59:05.459370 8385 raft_consensus.cc:2781] T b284ed2cb0264ced99a4ffd90983d902 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:05.459444 8385 raft_consensus.cc:686] T b284ed2cb0264ced99a4ffd90983d902 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 LEADER]: Becoming Leader. State: Replica: 1f9cb72be98941b5a8e504cbe34132b9, State: Running, Role: LEADER
I20240501 07:59:05.459563 8385 consensus_queue.cc:237] T b284ed2cb0264ced99a4ffd90983d902 P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:59:05.461712 6166 catalog_manager.cc:5484] T b284ed2cb0264ced99a4ffd90983d902 P 1f9cb72be98941b5a8e504cbe34132b9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132). New cstate: current_term: 1 leader_uuid: "1f9cb72be98941b5a8e504cbe34132b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } health_report { overall_health: HEALTHY } } }
I20240501 07:59:05.565124 8447 raft_consensus.cc:482] T aa1b8716228b4fd9af95870b7a613fb2 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:05.565243 8447 raft_consensus.cc:504] T aa1b8716228b4fd9af95870b7a613fb2 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } }
I20240501 07:59:05.565601 8447 leader_election.cc:288] T aa1b8716228b4fd9af95870b7a613fb2 P 8b8b3ad3aa8f4053b2167dd550f2086f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132:36773), 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130:32941)
I20240501 07:59:05.566042 6708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa1b8716228b4fd9af95870b7a613fb2" candidate_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f9cb72be98941b5a8e504cbe34132b9" is_pre_election: true
I20240501 07:59:05.566197 6708 raft_consensus.cc:2454] T aa1b8716228b4fd9af95870b7a613fb2 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b8b3ad3aa8f4053b2167dd550f2086f in term 0.
I20240501 07:59:05.566079 6444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa1b8716228b4fd9af95870b7a613fb2" candidate_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8315d22c56f7439a9e1fc82f153f018c" is_pre_election: true
I20240501 07:59:05.566509 6444 raft_consensus.cc:2454] T aa1b8716228b4fd9af95870b7a613fb2 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b8b3ad3aa8f4053b2167dd550f2086f in term 0.
I20240501 07:59:05.566494 6250 leader_election.cc:302] T aa1b8716228b4fd9af95870b7a613fb2 P 8b8b3ad3aa8f4053b2167dd550f2086f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f9cb72be98941b5a8e504cbe34132b9, 8b8b3ad3aa8f4053b2167dd550f2086f; no voters:
I20240501 07:59:05.566717 8447 raft_consensus.cc:2781] T aa1b8716228b4fd9af95870b7a613fb2 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:05.566788 8447 raft_consensus.cc:482] T aa1b8716228b4fd9af95870b7a613fb2 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:05.566843 8447 raft_consensus.cc:3037] T aa1b8716228b4fd9af95870b7a613fb2 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:05.567709 8447 raft_consensus.cc:504] T aa1b8716228b4fd9af95870b7a613fb2 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } }
I20240501 07:59:05.568061 8447 leader_election.cc:288] T aa1b8716228b4fd9af95870b7a613fb2 P 8b8b3ad3aa8f4053b2167dd550f2086f [CANDIDATE]: Term 1 election: Requested vote from peers 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132:36773), 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130:32941)
I20240501 07:59:05.568485 6444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa1b8716228b4fd9af95870b7a613fb2" candidate_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8315d22c56f7439a9e1fc82f153f018c"
I20240501 07:59:05.568595 6444 raft_consensus.cc:3037] T aa1b8716228b4fd9af95870b7a613fb2 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:05.568547 6708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa1b8716228b4fd9af95870b7a613fb2" candidate_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f9cb72be98941b5a8e504cbe34132b9"
I20240501 07:59:05.568675 6708 raft_consensus.cc:3037] T aa1b8716228b4fd9af95870b7a613fb2 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:05.569634 6444 raft_consensus.cc:2454] T aa1b8716228b4fd9af95870b7a613fb2 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b8b3ad3aa8f4053b2167dd550f2086f in term 1.
I20240501 07:59:05.569700 6708 raft_consensus.cc:2454] T aa1b8716228b4fd9af95870b7a613fb2 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b8b3ad3aa8f4053b2167dd550f2086f in term 1.
I20240501 07:59:05.569888 6250 leader_election.cc:302] T aa1b8716228b4fd9af95870b7a613fb2 P 8b8b3ad3aa8f4053b2167dd550f2086f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8315d22c56f7439a9e1fc82f153f018c, 8b8b3ad3aa8f4053b2167dd550f2086f; no voters:
I20240501 07:59:05.570051 8447 raft_consensus.cc:2781] T aa1b8716228b4fd9af95870b7a613fb2 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:05.570150 8447 raft_consensus.cc:686] T aa1b8716228b4fd9af95870b7a613fb2 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 LEADER]: Becoming Leader. State: Replica: 8b8b3ad3aa8f4053b2167dd550f2086f, State: Running, Role: LEADER
I20240501 07:59:05.570281 8447 consensus_queue.cc:237] T aa1b8716228b4fd9af95870b7a613fb2 P 8b8b3ad3aa8f4053b2167dd550f2086f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } }
I20240501 07:59:05.572177 6166 catalog_manager.cc:5484] T aa1b8716228b4fd9af95870b7a613fb2 P 8b8b3ad3aa8f4053b2167dd550f2086f reported cstate change: term changed from 0 to 1, leader changed from <none> to 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129). New cstate: current_term: 1 leader_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } health_report { overall_health: UNKNOWN } } }
W20240501 07:59:05.750792 8592 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:59:05.751057 8592 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:59:05.754300 8592 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_18'
I20240501 07:59:05.838707 8385 consensus_queue.cc:1035] T ae46990350284295bf14aa64d4b6c93e P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:05.846681 8385 consensus_queue.cc:1035] T ae46990350284295bf14aa64d4b6c93e P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:05.856986 6707 raft_consensus.cc:1261] T aa1b8716228b4fd9af95870b7a613fb2 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Refusing update from remote peer 8b8b3ad3aa8f4053b2167dd550f2086f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:05.856992 6445 raft_consensus.cc:1261] T aa1b8716228b4fd9af95870b7a613fb2 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Refusing update from remote peer 8b8b3ad3aa8f4053b2167dd550f2086f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:05.857570 8615 consensus_queue.cc:1035] T aa1b8716228b4fd9af95870b7a613fb2 P 8b8b3ad3aa8f4053b2167dd550f2086f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:05.857710 8105 consensus_queue.cc:1035] T aa1b8716228b4fd9af95870b7a613fb2 P 8b8b3ad3aa8f4053b2167dd550f2086f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:05.871311 6577 raft_consensus.cc:1261] T b284ed2cb0264ced99a4ffd90983d902 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Refusing update from remote peer 1f9cb72be98941b5a8e504cbe34132b9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:05.871742 8385 consensus_queue.cc:1035] T b284ed2cb0264ced99a4ffd90983d902 P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:05.875156 6314 raft_consensus.cc:1261] T b284ed2cb0264ced99a4ffd90983d902 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Refusing update from remote peer 1f9cb72be98941b5a8e504cbe34132b9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:05.875566 8385 consensus_queue.cc:1035] T b284ed2cb0264ced99a4ffd90983d902 P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 129.687 ms
time per row: 0.0648437 ms
I20240501 07:59:05.934911 6169 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55950:
name: "rebalance_test_table_19"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 07:59:05.940858 6294 tablet_service.cc:1467] Processing CreateTablet for tablet 2398430a23094711a13a86cc88e27a78 (DEFAULT_TABLE table=rebalance_test_table_19 [id=35f40c7e6b1e4d799576a7d98130d848]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:05.941032 6557 tablet_service.cc:1467] Processing CreateTablet for tablet 6c3c0d6fe41f4ae196ae29beb9b68125 (DEFAULT_TABLE table=rebalance_test_table_19 [id=35f40c7e6b1e4d799576a7d98130d848]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:05.941285 6294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2398430a23094711a13a86cc88e27a78. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:05.941320 6557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c3c0d6fe41f4ae196ae29beb9b68125. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:05.941570 6555 tablet_service.cc:1467] Processing CreateTablet for tablet 404bb81b9c404dd288d2f8160a4273da (DEFAULT_TABLE table=rebalance_test_table_19 [id=35f40c7e6b1e4d799576a7d98130d848]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:05.941809 6555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 404bb81b9c404dd288d2f8160a4273da. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:05.943194 8653 tablet_bootstrap.cc:492] T 6c3c0d6fe41f4ae196ae29beb9b68125 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Bootstrap starting.
I20240501 07:59:05.943318 6556 tablet_service.cc:1467] Processing CreateTablet for tablet 2398430a23094711a13a86cc88e27a78 (DEFAULT_TABLE table=rebalance_test_table_19 [id=35f40c7e6b1e4d799576a7d98130d848]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:05.943506 6556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2398430a23094711a13a86cc88e27a78. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:05.944206 8653 tablet_bootstrap.cc:654] T 6c3c0d6fe41f4ae196ae29beb9b68125 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:05.944279 8654 tablet_bootstrap.cc:492] T 2398430a23094711a13a86cc88e27a78 P 8b8b3ad3aa8f4053b2167dd550f2086f: Bootstrap starting.
I20240501 07:59:05.945190 8654 tablet_bootstrap.cc:654] T 2398430a23094711a13a86cc88e27a78 P 8b8b3ad3aa8f4053b2167dd550f2086f: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:05.945887 6426 tablet_service.cc:1467] Processing CreateTablet for tablet 2398430a23094711a13a86cc88e27a78 (DEFAULT_TABLE table=rebalance_test_table_19 [id=35f40c7e6b1e4d799576a7d98130d848]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:05.946170 6426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2398430a23094711a13a86cc88e27a78. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:05.946151 6687 tablet_service.cc:1467] Processing CreateTablet for tablet 6c3c0d6fe41f4ae196ae29beb9b68125 (DEFAULT_TABLE table=rebalance_test_table_19 [id=35f40c7e6b1e4d799576a7d98130d848]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:05.946223 6425 tablet_service.cc:1467] Processing CreateTablet for tablet 404bb81b9c404dd288d2f8160a4273da (DEFAULT_TABLE table=rebalance_test_table_19 [id=35f40c7e6b1e4d799576a7d98130d848]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:05.946401 6687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c3c0d6fe41f4ae196ae29beb9b68125. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:05.946458 6425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 404bb81b9c404dd288d2f8160a4273da. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:05.946863 8653 tablet_bootstrap.cc:492] T 6c3c0d6fe41f4ae196ae29beb9b68125 P 3b164cbbc00f4e8eb3254ddd349a3ff9: No bootstrap required, opened a new log
I20240501 07:59:05.946961 8653 ts_tablet_manager.cc:1367] T 6c3c0d6fe41f4ae196ae29beb9b68125 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240501 07:59:05.947415 8653 raft_consensus.cc:348] T 6c3c0d6fe41f4ae196ae29beb9b68125 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } }
I20240501 07:59:05.947536 8653 raft_consensus.cc:374] T 6c3c0d6fe41f4ae196ae29beb9b68125 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:05.947571 8653 raft_consensus.cc:729] T 6c3c0d6fe41f4ae196ae29beb9b68125 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b164cbbc00f4e8eb3254ddd349a3ff9, State: Initialized, Role: FOLLOWER
I20240501 07:59:05.947727 8653 consensus_queue.cc:260] T 6c3c0d6fe41f4ae196ae29beb9b68125 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } }
I20240501 07:59:05.948784 6686 tablet_service.cc:1467] Processing CreateTablet for tablet 404bb81b9c404dd288d2f8160a4273da (DEFAULT_TABLE table=rebalance_test_table_19 [id=35f40c7e6b1e4d799576a7d98130d848]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:05.949040 6686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 404bb81b9c404dd288d2f8160a4273da. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:05.949083 8658 tablet_bootstrap.cc:492] T 6c3c0d6fe41f4ae196ae29beb9b68125 P 1f9cb72be98941b5a8e504cbe34132b9: Bootstrap starting.
I20240501 07:59:05.949230 8653 ts_tablet_manager.cc:1398] T 6c3c0d6fe41f4ae196ae29beb9b68125 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:59:05.949435 8653 tablet_bootstrap.cc:492] T 404bb81b9c404dd288d2f8160a4273da P 3b164cbbc00f4e8eb3254ddd349a3ff9: Bootstrap starting.
I20240501 07:59:05.949455 8657 tablet_bootstrap.cc:492] T 2398430a23094711a13a86cc88e27a78 P 8315d22c56f7439a9e1fc82f153f018c: Bootstrap starting.
I20240501 07:59:05.949862 6294 tablet_service.cc:1467] Processing CreateTablet for tablet 6c3c0d6fe41f4ae196ae29beb9b68125 (DEFAULT_TABLE table=rebalance_test_table_19 [id=35f40c7e6b1e4d799576a7d98130d848]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:05.950147 6294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c3c0d6fe41f4ae196ae29beb9b68125. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:05.950356 8653 tablet_bootstrap.cc:654] T 404bb81b9c404dd288d2f8160a4273da P 3b164cbbc00f4e8eb3254ddd349a3ff9: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:05.951429 8653 tablet_bootstrap.cc:492] T 404bb81b9c404dd288d2f8160a4273da P 3b164cbbc00f4e8eb3254ddd349a3ff9: No bootstrap required, opened a new log
I20240501 07:59:05.951505 8653 ts_tablet_manager.cc:1367] T 404bb81b9c404dd288d2f8160a4273da P 3b164cbbc00f4e8eb3254ddd349a3ff9: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 07:59:05.951936 8653 raft_consensus.cc:348] T 404bb81b9c404dd288d2f8160a4273da P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } }
I20240501 07:59:05.952092 8653 raft_consensus.cc:374] T 404bb81b9c404dd288d2f8160a4273da P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:05.952127 8653 raft_consensus.cc:729] T 404bb81b9c404dd288d2f8160a4273da P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b164cbbc00f4e8eb3254ddd349a3ff9, State: Initialized, Role: FOLLOWER
I20240501 07:59:05.952242 8653 consensus_queue.cc:260] T 404bb81b9c404dd288d2f8160a4273da P 3b164cbbc00f4e8eb3254ddd349a3ff9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } }
I20240501 07:59:05.952530 8653 ts_tablet_manager.cc:1398] T 404bb81b9c404dd288d2f8160a4273da P 3b164cbbc00f4e8eb3254ddd349a3ff9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:05.952685 8653 tablet_bootstrap.cc:492] T 2398430a23094711a13a86cc88e27a78 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Bootstrap starting.
I20240501 07:59:05.952769 8657 tablet_bootstrap.cc:654] T 2398430a23094711a13a86cc88e27a78 P 8315d22c56f7439a9e1fc82f153f018c: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:05.952826 8658 tablet_bootstrap.cc:654] T 6c3c0d6fe41f4ae196ae29beb9b68125 P 1f9cb72be98941b5a8e504cbe34132b9: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:05.953629 8653 tablet_bootstrap.cc:654] T 2398430a23094711a13a86cc88e27a78 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:05.954352 8658 tablet_bootstrap.cc:492] T 6c3c0d6fe41f4ae196ae29beb9b68125 P 1f9cb72be98941b5a8e504cbe34132b9: No bootstrap required, opened a new log
I20240501 07:59:05.954465 8658 ts_tablet_manager.cc:1367] T 6c3c0d6fe41f4ae196ae29beb9b68125 P 1f9cb72be98941b5a8e504cbe34132b9: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.001s
I20240501 07:59:05.954643 8654 tablet_bootstrap.cc:492] T 2398430a23094711a13a86cc88e27a78 P 8b8b3ad3aa8f4053b2167dd550f2086f: No bootstrap required, opened a new log
I20240501 07:59:05.954674 8653 tablet_bootstrap.cc:492] T 2398430a23094711a13a86cc88e27a78 P 3b164cbbc00f4e8eb3254ddd349a3ff9: No bootstrap required, opened a new log
I20240501 07:59:05.954749 8654 ts_tablet_manager.cc:1367] T 2398430a23094711a13a86cc88e27a78 P 8b8b3ad3aa8f4053b2167dd550f2086f: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240501 07:59:05.954777 8653 ts_tablet_manager.cc:1367] T 2398430a23094711a13a86cc88e27a78 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 07:59:05.954963 8658 raft_consensus.cc:348] T 6c3c0d6fe41f4ae196ae29beb9b68125 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } }
I20240501 07:59:05.955129 8658 raft_consensus.cc:374] T 6c3c0d6fe41f4ae196ae29beb9b68125 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:05.955189 8658 raft_consensus.cc:729] T 6c3c0d6fe41f4ae196ae29beb9b68125 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f9cb72be98941b5a8e504cbe34132b9, State: Initialized, Role: FOLLOWER
I20240501 07:59:05.955317 8654 raft_consensus.cc:348] T 2398430a23094711a13a86cc88e27a78 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } }
I20240501 07:59:05.955338 8658 consensus_queue.cc:260] T 6c3c0d6fe41f4ae196ae29beb9b68125 P 1f9cb72be98941b5a8e504cbe34132b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } }
I20240501 07:59:05.955441 8654 raft_consensus.cc:374] T 2398430a23094711a13a86cc88e27a78 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:05.955396 8653 raft_consensus.cc:348] T 2398430a23094711a13a86cc88e27a78 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } }
I20240501 07:59:05.955487 8653 raft_consensus.cc:374] T 2398430a23094711a13a86cc88e27a78 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:05.955494 8654 raft_consensus.cc:729] T 2398430a23094711a13a86cc88e27a78 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b8b3ad3aa8f4053b2167dd550f2086f, State: Initialized, Role: FOLLOWER
I20240501 07:59:05.955530 8653 raft_consensus.cc:729] T 2398430a23094711a13a86cc88e27a78 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b164cbbc00f4e8eb3254ddd349a3ff9, State: Initialized, Role: FOLLOWER
I20240501 07:59:05.955642 8658 ts_tablet_manager.cc:1398] T 6c3c0d6fe41f4ae196ae29beb9b68125 P 1f9cb72be98941b5a8e504cbe34132b9: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:59:05.955667 8654 consensus_queue.cc:260] T 2398430a23094711a13a86cc88e27a78 P 8b8b3ad3aa8f4053b2167dd550f2086f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } }
I20240501 07:59:05.955677 8653 consensus_queue.cc:260] T 2398430a23094711a13a86cc88e27a78 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } }
I20240501 07:59:05.956422 8658 tablet_bootstrap.cc:492] T 404bb81b9c404dd288d2f8160a4273da P 1f9cb72be98941b5a8e504cbe34132b9: Bootstrap starting.
I20240501 07:59:05.956670 8653 ts_tablet_manager.cc:1398] T 2398430a23094711a13a86cc88e27a78 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:59:05.957208 8654 ts_tablet_manager.cc:1398] T 2398430a23094711a13a86cc88e27a78 P 8b8b3ad3aa8f4053b2167dd550f2086f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:59:05.957367 8658 tablet_bootstrap.cc:654] T 404bb81b9c404dd288d2f8160a4273da P 1f9cb72be98941b5a8e504cbe34132b9: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:05.957417 8654 tablet_bootstrap.cc:492] T 6c3c0d6fe41f4ae196ae29beb9b68125 P 8b8b3ad3aa8f4053b2167dd550f2086f: Bootstrap starting.
I20240501 07:59:05.958782 8658 tablet_bootstrap.cc:492] T 404bb81b9c404dd288d2f8160a4273da P 1f9cb72be98941b5a8e504cbe34132b9: No bootstrap required, opened a new log
I20240501 07:59:05.958945 8658 ts_tablet_manager.cc:1367] T 404bb81b9c404dd288d2f8160a4273da P 1f9cb72be98941b5a8e504cbe34132b9: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 07:59:05.959443 8658 raft_consensus.cc:348] T 404bb81b9c404dd288d2f8160a4273da P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } }
I20240501 07:59:05.959610 8658 raft_consensus.cc:374] T 404bb81b9c404dd288d2f8160a4273da P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:05.959688 8658 raft_consensus.cc:729] T 404bb81b9c404dd288d2f8160a4273da P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f9cb72be98941b5a8e504cbe34132b9, State: Initialized, Role: FOLLOWER
I20240501 07:59:05.959702 8654 tablet_bootstrap.cc:654] T 6c3c0d6fe41f4ae196ae29beb9b68125 P 8b8b3ad3aa8f4053b2167dd550f2086f: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:05.959883 8658 consensus_queue.cc:260] T 404bb81b9c404dd288d2f8160a4273da P 1f9cb72be98941b5a8e504cbe34132b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } }
I20240501 07:59:05.960294 8658 ts_tablet_manager.cc:1398] T 404bb81b9c404dd288d2f8160a4273da P 1f9cb72be98941b5a8e504cbe34132b9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:05.960796 8657 tablet_bootstrap.cc:492] T 2398430a23094711a13a86cc88e27a78 P 8315d22c56f7439a9e1fc82f153f018c: No bootstrap required, opened a new log
I20240501 07:59:05.960928 8657 ts_tablet_manager.cc:1367] T 2398430a23094711a13a86cc88e27a78 P 8315d22c56f7439a9e1fc82f153f018c: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.001s
I20240501 07:59:05.961457 8657 raft_consensus.cc:348] T 2398430a23094711a13a86cc88e27a78 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } }
I20240501 07:59:05.961575 8657 raft_consensus.cc:374] T 2398430a23094711a13a86cc88e27a78 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:05.961606 8657 raft_consensus.cc:729] T 2398430a23094711a13a86cc88e27a78 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8315d22c56f7439a9e1fc82f153f018c, State: Initialized, Role: FOLLOWER
I20240501 07:59:05.961724 8657 consensus_queue.cc:260] T 2398430a23094711a13a86cc88e27a78 P 8315d22c56f7439a9e1fc82f153f018c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } }
I20240501 07:59:05.962102 8657 ts_tablet_manager.cc:1398] T 2398430a23094711a13a86cc88e27a78 P 8315d22c56f7439a9e1fc82f153f018c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:05.962288 8654 tablet_bootstrap.cc:492] T 6c3c0d6fe41f4ae196ae29beb9b68125 P 8b8b3ad3aa8f4053b2167dd550f2086f: No bootstrap required, opened a new log
I20240501 07:59:05.962422 8654 ts_tablet_manager.cc:1367] T 6c3c0d6fe41f4ae196ae29beb9b68125 P 8b8b3ad3aa8f4053b2167dd550f2086f: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 07:59:05.962432 8657 tablet_bootstrap.cc:492] T 404bb81b9c404dd288d2f8160a4273da P 8315d22c56f7439a9e1fc82f153f018c: Bootstrap starting.
I20240501 07:59:05.963162 8654 raft_consensus.cc:348] T 6c3c0d6fe41f4ae196ae29beb9b68125 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } }
I20240501 07:59:05.963339 8654 raft_consensus.cc:374] T 6c3c0d6fe41f4ae196ae29beb9b68125 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:05.963433 8654 raft_consensus.cc:729] T 6c3c0d6fe41f4ae196ae29beb9b68125 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b8b3ad3aa8f4053b2167dd550f2086f, State: Initialized, Role: FOLLOWER
I20240501 07:59:05.963348 8657 tablet_bootstrap.cc:654] T 404bb81b9c404dd288d2f8160a4273da P 8315d22c56f7439a9e1fc82f153f018c: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:05.963640 8654 consensus_queue.cc:260] T 6c3c0d6fe41f4ae196ae29beb9b68125 P 8b8b3ad3aa8f4053b2167dd550f2086f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } }
I20240501 07:59:05.964175 8654 ts_tablet_manager.cc:1398] T 6c3c0d6fe41f4ae196ae29beb9b68125 P 8b8b3ad3aa8f4053b2167dd550f2086f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:59:05.965552 8657 tablet_bootstrap.cc:492] T 404bb81b9c404dd288d2f8160a4273da P 8315d22c56f7439a9e1fc82f153f018c: No bootstrap required, opened a new log
I20240501 07:59:05.965715 8657 ts_tablet_manager.cc:1367] T 404bb81b9c404dd288d2f8160a4273da P 8315d22c56f7439a9e1fc82f153f018c: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20240501 07:59:05.966077 8657 raft_consensus.cc:348] T 404bb81b9c404dd288d2f8160a4273da P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } }
I20240501 07:59:05.966162 8657 raft_consensus.cc:374] T 404bb81b9c404dd288d2f8160a4273da P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:05.966195 8657 raft_consensus.cc:729] T 404bb81b9c404dd288d2f8160a4273da P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8315d22c56f7439a9e1fc82f153f018c, State: Initialized, Role: FOLLOWER
I20240501 07:59:05.966341 8657 consensus_queue.cc:260] T 404bb81b9c404dd288d2f8160a4273da P 8315d22c56f7439a9e1fc82f153f018c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } }
I20240501 07:59:05.967265 8657 ts_tablet_manager.cc:1398] T 404bb81b9c404dd288d2f8160a4273da P 8315d22c56f7439a9e1fc82f153f018c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:59:05.977293 8105 raft_consensus.cc:482] T 6c3c0d6fe41f4ae196ae29beb9b68125 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:05.977420 8105 raft_consensus.cc:504] T 6c3c0d6fe41f4ae196ae29beb9b68125 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } }
I20240501 07:59:05.977906 8105 leader_election.cc:288] T 6c3c0d6fe41f4ae196ae29beb9b68125 P 8b8b3ad3aa8f4053b2167dd550f2086f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131:34471), 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132:36773)
I20240501 07:59:05.978227 6707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c3c0d6fe41f4ae196ae29beb9b68125" candidate_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f9cb72be98941b5a8e504cbe34132b9" is_pre_election: true
I20240501 07:59:05.978219 6577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c3c0d6fe41f4ae196ae29beb9b68125" candidate_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" is_pre_election: true
I20240501 07:59:05.978359 6707 raft_consensus.cc:2454] T 6c3c0d6fe41f4ae196ae29beb9b68125 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b8b3ad3aa8f4053b2167dd550f2086f in term 0.
I20240501 07:59:05.978380 6577 raft_consensus.cc:2454] T 6c3c0d6fe41f4ae196ae29beb9b68125 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b8b3ad3aa8f4053b2167dd550f2086f in term 0.
I20240501 07:59:05.978652 6250 leader_election.cc:302] T 6c3c0d6fe41f4ae196ae29beb9b68125 P 8b8b3ad3aa8f4053b2167dd550f2086f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f9cb72be98941b5a8e504cbe34132b9, 8b8b3ad3aa8f4053b2167dd550f2086f; no voters:
I20240501 07:59:05.978844 8105 raft_consensus.cc:2781] T 6c3c0d6fe41f4ae196ae29beb9b68125 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:05.978902 8105 raft_consensus.cc:482] T 6c3c0d6fe41f4ae196ae29beb9b68125 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:05.978952 8105 raft_consensus.cc:3037] T 6c3c0d6fe41f4ae196ae29beb9b68125 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:05.979775 8105 raft_consensus.cc:504] T 6c3c0d6fe41f4ae196ae29beb9b68125 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } }
I20240501 07:59:05.980084 8105 leader_election.cc:288] T 6c3c0d6fe41f4ae196ae29beb9b68125 P 8b8b3ad3aa8f4053b2167dd550f2086f [CANDIDATE]: Term 1 election: Requested vote from peers 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131:34471), 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132:36773)
I20240501 07:59:05.980393 6707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c3c0d6fe41f4ae196ae29beb9b68125" candidate_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f9cb72be98941b5a8e504cbe34132b9"
I20240501 07:59:05.980506 6707 raft_consensus.cc:3037] T 6c3c0d6fe41f4ae196ae29beb9b68125 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:05.980482 6577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c3c0d6fe41f4ae196ae29beb9b68125" candidate_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9"
I20240501 07:59:05.980598 6577 raft_consensus.cc:3037] T 6c3c0d6fe41f4ae196ae29beb9b68125 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:05.981505 6707 raft_consensus.cc:2454] T 6c3c0d6fe41f4ae196ae29beb9b68125 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b8b3ad3aa8f4053b2167dd550f2086f in term 1.
I20240501 07:59:05.981504 6577 raft_consensus.cc:2454] T 6c3c0d6fe41f4ae196ae29beb9b68125 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b8b3ad3aa8f4053b2167dd550f2086f in term 1.
I20240501 07:59:05.981793 6250 leader_election.cc:302] T 6c3c0d6fe41f4ae196ae29beb9b68125 P 8b8b3ad3aa8f4053b2167dd550f2086f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f9cb72be98941b5a8e504cbe34132b9, 8b8b3ad3aa8f4053b2167dd550f2086f; no voters:
I20240501 07:59:05.981977 8105 raft_consensus.cc:2781] T 6c3c0d6fe41f4ae196ae29beb9b68125 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:05.982079 8105 raft_consensus.cc:686] T 6c3c0d6fe41f4ae196ae29beb9b68125 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 LEADER]: Becoming Leader. State: Replica: 8b8b3ad3aa8f4053b2167dd550f2086f, State: Running, Role: LEADER
I20240501 07:59:05.982229 8105 consensus_queue.cc:237] T 6c3c0d6fe41f4ae196ae29beb9b68125 P 8b8b3ad3aa8f4053b2167dd550f2086f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } }
I20240501 07:59:05.984169 6172 catalog_manager.cc:5484] T 6c3c0d6fe41f4ae196ae29beb9b68125 P 8b8b3ad3aa8f4053b2167dd550f2086f reported cstate change: term changed from 0 to 1, leader changed from <none> to 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129). New cstate: current_term: 1 leader_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } health_report { overall_health: HEALTHY } } }
I20240501 07:59:06.003051 8636 raft_consensus.cc:482] T 404bb81b9c404dd288d2f8160a4273da P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:06.003197 8636 raft_consensus.cc:504] T 404bb81b9c404dd288d2f8160a4273da P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } }
I20240501 07:59:06.003618 8636 leader_election.cc:288] T 404bb81b9c404dd288d2f8160a4273da P 1f9cb72be98941b5a8e504cbe34132b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130:32941), 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131:34471)
I20240501 07:59:06.003971 6577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "404bb81b9c404dd288d2f8160a4273da" candidate_uuid: "1f9cb72be98941b5a8e504cbe34132b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" is_pre_election: true
I20240501 07:59:06.004113 6577 raft_consensus.cc:2454] T 404bb81b9c404dd288d2f8160a4273da P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f9cb72be98941b5a8e504cbe34132b9 in term 0.
I20240501 07:59:06.004091 6444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "404bb81b9c404dd288d2f8160a4273da" candidate_uuid: "1f9cb72be98941b5a8e504cbe34132b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8315d22c56f7439a9e1fc82f153f018c" is_pre_election: true
I20240501 07:59:06.004206 6444 raft_consensus.cc:2454] T 404bb81b9c404dd288d2f8160a4273da P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f9cb72be98941b5a8e504cbe34132b9 in term 0.
I20240501 07:59:06.004298 8634 raft_consensus.cc:482] T 404bb81b9c404dd288d2f8160a4273da P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:06.004417 8634 raft_consensus.cc:504] T 404bb81b9c404dd288d2f8160a4273da P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } }
I20240501 07:59:06.004523 6643 leader_election.cc:302] T 404bb81b9c404dd288d2f8160a4273da P 1f9cb72be98941b5a8e504cbe34132b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f9cb72be98941b5a8e504cbe34132b9, 8315d22c56f7439a9e1fc82f153f018c; no voters:
I20240501 07:59:06.004693 8636 raft_consensus.cc:2781] T 404bb81b9c404dd288d2f8160a4273da P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:06.004752 8636 raft_consensus.cc:482] T 404bb81b9c404dd288d2f8160a4273da P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:06.004799 8636 raft_consensus.cc:3037] T 404bb81b9c404dd288d2f8160a4273da P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:06.004817 8634 leader_election.cc:288] T 404bb81b9c404dd288d2f8160a4273da P 8315d22c56f7439a9e1fc82f153f018c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132:36773), 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131:34471)
I20240501 07:59:06.005230 6707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "404bb81b9c404dd288d2f8160a4273da" candidate_uuid: "8315d22c56f7439a9e1fc82f153f018c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f9cb72be98941b5a8e504cbe34132b9" is_pre_election: true
I20240501 07:59:06.005340 6577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "404bb81b9c404dd288d2f8160a4273da" candidate_uuid: "8315d22c56f7439a9e1fc82f153f018c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" is_pre_election: true
I20240501 07:59:06.005450 6577 raft_consensus.cc:2454] T 404bb81b9c404dd288d2f8160a4273da P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8315d22c56f7439a9e1fc82f153f018c in term 0.
I20240501 07:59:06.005728 6382 leader_election.cc:302] T 404bb81b9c404dd288d2f8160a4273da P 8315d22c56f7439a9e1fc82f153f018c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b164cbbc00f4e8eb3254ddd349a3ff9, 8315d22c56f7439a9e1fc82f153f018c; no voters:
I20240501 07:59:06.006099 8636 raft_consensus.cc:504] T 404bb81b9c404dd288d2f8160a4273da P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } }
I20240501 07:59:06.006292 6707 raft_consensus.cc:2379] T 404bb81b9c404dd288d2f8160a4273da P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8315d22c56f7439a9e1fc82f153f018c in current term 1: Already voted for candidate 1f9cb72be98941b5a8e504cbe34132b9 in this term.
I20240501 07:59:06.006703 6444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "404bb81b9c404dd288d2f8160a4273da" candidate_uuid: "1f9cb72be98941b5a8e504cbe34132b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8315d22c56f7439a9e1fc82f153f018c"
I20240501 07:59:06.006765 6577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "404bb81b9c404dd288d2f8160a4273da" candidate_uuid: "1f9cb72be98941b5a8e504cbe34132b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9"
I20240501 07:59:06.006803 6444 raft_consensus.cc:3037] T 404bb81b9c404dd288d2f8160a4273da P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:06.006860 6577 raft_consensus.cc:3037] T 404bb81b9c404dd288d2f8160a4273da P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:06.007647 6577 raft_consensus.cc:2454] T 404bb81b9c404dd288d2f8160a4273da P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f9cb72be98941b5a8e504cbe34132b9 in term 1.
I20240501 07:59:06.007774 6444 raft_consensus.cc:2454] T 404bb81b9c404dd288d2f8160a4273da P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f9cb72be98941b5a8e504cbe34132b9 in term 1.
I20240501 07:59:06.007895 6644 leader_election.cc:302] T 404bb81b9c404dd288d2f8160a4273da P 1f9cb72be98941b5a8e504cbe34132b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f9cb72be98941b5a8e504cbe34132b9, 3b164cbbc00f4e8eb3254ddd349a3ff9; no voters:
I20240501 07:59:06.007778 8636 leader_election.cc:288] T 404bb81b9c404dd288d2f8160a4273da P 1f9cb72be98941b5a8e504cbe34132b9 [CANDIDATE]: Term 1 election: Requested vote from peers 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130:32941), 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131:34471)
I20240501 07:59:06.008033 8385 raft_consensus.cc:2781] T 404bb81b9c404dd288d2f8160a4273da P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:06.008112 8385 raft_consensus.cc:686] T 404bb81b9c404dd288d2f8160a4273da P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 LEADER]: Becoming Leader. State: Replica: 1f9cb72be98941b5a8e504cbe34132b9, State: Running, Role: LEADER
I20240501 07:59:06.008271 8385 consensus_queue.cc:237] T 404bb81b9c404dd288d2f8160a4273da P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } }
I20240501 07:59:06.008858 8634 raft_consensus.cc:2741] T 404bb81b9c404dd288d2f8160a4273da P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20240501 07:59:06.010550 6172 catalog_manager.cc:5484] T 404bb81b9c404dd288d2f8160a4273da P 1f9cb72be98941b5a8e504cbe34132b9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132). New cstate: current_term: 1 leader_uuid: "1f9cb72be98941b5a8e504cbe34132b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } health_report { overall_health: UNKNOWN } } }
I20240501 07:59:06.140208 8447 raft_consensus.cc:482] T 2398430a23094711a13a86cc88e27a78 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:06.140362 8447 raft_consensus.cc:504] T 2398430a23094711a13a86cc88e27a78 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } }
I20240501 07:59:06.140803 8447 leader_election.cc:288] T 2398430a23094711a13a86cc88e27a78 P 8b8b3ad3aa8f4053b2167dd550f2086f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131:34471), 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130:32941)
I20240501 07:59:06.141156 6577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2398430a23094711a13a86cc88e27a78" candidate_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" is_pre_election: true
I20240501 07:59:06.141183 6444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2398430a23094711a13a86cc88e27a78" candidate_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8315d22c56f7439a9e1fc82f153f018c" is_pre_election: true
I20240501 07:59:06.141304 6577 raft_consensus.cc:2454] T 2398430a23094711a13a86cc88e27a78 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b8b3ad3aa8f4053b2167dd550f2086f in term 0.
I20240501 07:59:06.141340 6444 raft_consensus.cc:2454] T 2398430a23094711a13a86cc88e27a78 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b8b3ad3aa8f4053b2167dd550f2086f in term 0.
I20240501 07:59:06.141613 6250 leader_election.cc:302] T 2398430a23094711a13a86cc88e27a78 P 8b8b3ad3aa8f4053b2167dd550f2086f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8315d22c56f7439a9e1fc82f153f018c, 8b8b3ad3aa8f4053b2167dd550f2086f; no voters:
I20240501 07:59:06.141758 8447 raft_consensus.cc:2781] T 2398430a23094711a13a86cc88e27a78 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:06.141811 8447 raft_consensus.cc:482] T 2398430a23094711a13a86cc88e27a78 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:06.141855 8447 raft_consensus.cc:3037] T 2398430a23094711a13a86cc88e27a78 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:06.142707 8447 raft_consensus.cc:504] T 2398430a23094711a13a86cc88e27a78 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } }
I20240501 07:59:06.142988 8447 leader_election.cc:288] T 2398430a23094711a13a86cc88e27a78 P 8b8b3ad3aa8f4053b2167dd550f2086f [CANDIDATE]: Term 1 election: Requested vote from peers 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131:34471), 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130:32941)
I20240501 07:59:06.143298 6577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2398430a23094711a13a86cc88e27a78" candidate_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9"
I20240501 07:59:06.143309 6444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2398430a23094711a13a86cc88e27a78" candidate_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8315d22c56f7439a9e1fc82f153f018c"
I20240501 07:59:06.143396 6577 raft_consensus.cc:3037] T 2398430a23094711a13a86cc88e27a78 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:06.143422 6444 raft_consensus.cc:3037] T 2398430a23094711a13a86cc88e27a78 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:06.144308 6444 raft_consensus.cc:2454] T 2398430a23094711a13a86cc88e27a78 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b8b3ad3aa8f4053b2167dd550f2086f in term 1.
I20240501 07:59:06.144327 6577 raft_consensus.cc:2454] T 2398430a23094711a13a86cc88e27a78 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b8b3ad3aa8f4053b2167dd550f2086f in term 1.
I20240501 07:59:06.144589 6250 leader_election.cc:302] T 2398430a23094711a13a86cc88e27a78 P 8b8b3ad3aa8f4053b2167dd550f2086f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8315d22c56f7439a9e1fc82f153f018c, 8b8b3ad3aa8f4053b2167dd550f2086f; no voters:
I20240501 07:59:06.144909 8447 raft_consensus.cc:2781] T 2398430a23094711a13a86cc88e27a78 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:06.145018 8447 raft_consensus.cc:686] T 2398430a23094711a13a86cc88e27a78 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 LEADER]: Becoming Leader. State: Replica: 8b8b3ad3aa8f4053b2167dd550f2086f, State: Running, Role: LEADER
I20240501 07:59:06.145157 8447 consensus_queue.cc:237] T 2398430a23094711a13a86cc88e27a78 P 8b8b3ad3aa8f4053b2167dd550f2086f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } }
I20240501 07:59:06.147310 6172 catalog_manager.cc:5484] T 2398430a23094711a13a86cc88e27a78 P 8b8b3ad3aa8f4053b2167dd550f2086f reported cstate change: term changed from 0 to 1, leader changed from <none> to 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129). New cstate: current_term: 1 leader_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } health_report { overall_health: UNKNOWN } } }
W20240501 07:59:06.301302 8669 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:59:06.301563 8669 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:59:06.305234 8669 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_19'
I20240501 07:59:06.389390 6444 raft_consensus.cc:1261] T 2398430a23094711a13a86cc88e27a78 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Refusing update from remote peer 8b8b3ad3aa8f4053b2167dd550f2086f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:06.389750 6577 raft_consensus.cc:1261] T 2398430a23094711a13a86cc88e27a78 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Refusing update from remote peer 8b8b3ad3aa8f4053b2167dd550f2086f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:06.390070 8105 consensus_queue.cc:1035] T 2398430a23094711a13a86cc88e27a78 P 8b8b3ad3aa8f4053b2167dd550f2086f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:06.390971 8105 consensus_queue.cc:1035] T 2398430a23094711a13a86cc88e27a78 P 8b8b3ad3aa8f4053b2167dd550f2086f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:06.394457 6575 raft_consensus.cc:1261] T 6c3c0d6fe41f4ae196ae29beb9b68125 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Refusing update from remote peer 8b8b3ad3aa8f4053b2167dd550f2086f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:06.395074 8105 consensus_queue.cc:1035] T 6c3c0d6fe41f4ae196ae29beb9b68125 P 8b8b3ad3aa8f4053b2167dd550f2086f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:06.395323 6707 raft_consensus.cc:1261] T 6c3c0d6fe41f4ae196ae29beb9b68125 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Refusing update from remote peer 8b8b3ad3aa8f4053b2167dd550f2086f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:06.396065 8105 consensus_queue.cc:1035] T 6c3c0d6fe41f4ae196ae29beb9b68125 P 8b8b3ad3aa8f4053b2167dd550f2086f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:06.411866 6445 raft_consensus.cc:1261] T 404bb81b9c404dd288d2f8160a4273da P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Refusing update from remote peer 1f9cb72be98941b5a8e504cbe34132b9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:06.413934 8636 consensus_queue.cc:1035] T 404bb81b9c404dd288d2f8160a4273da P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:06.433437 6577 raft_consensus.cc:1261] T 404bb81b9c404dd288d2f8160a4273da P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Refusing update from remote peer 1f9cb72be98941b5a8e504cbe34132b9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:06.433912 8385 consensus_queue.cc:1035] T 404bb81b9c404dd288d2f8160a4273da P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 122.853 ms
time per row: 0.0614265 ms
I20240501 07:59:06.473650 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tbgEt/build/debug/bin/kudu
/tmp/dist-test-task7tbgEt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.133:39859
--local_ip_for_outbound_sockets=127.0.99.133
--tserver_master_addrs=127.0.99.190:38383
--webserver_port=37019
--webserver_interface=127.0.99.133
--builtin_ntp_servers=127.0.99.148:34871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240501 07:59:06.593526 8712 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:59:06.593755 8712 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 07:59:06.593825 8712 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:59:06.597116 8712 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 07:59:06.597208 8712 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 07:59:06.597309 8712 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.133
I20240501 07:59:06.601575 8712 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:34871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.133:39859
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.0.99.133
--webserver_port=37019
--tserver_master_addrs=127.0.99.190:38383
--never_fsync=true
--heap_profile_path=/tmp/kudu.8712
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.133
--log_dir=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 07:43:16 UTC on 1a15035a6830
build id 69
I20240501 07:59:06.602537 8712 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 07:59:06.603699 8712 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:59:06.613504 8718 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:59:06.614274 8720 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:59:06.615335 8717 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:59:06.616248 8712 server_base.cc:1034] running on GCE node
I20240501 07:59:06.616726 8712 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 07:59:06.617455 8712 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 07:59:06.618635 8712 hybrid_clock.cc:648] HybridClock initialized: now 1714550346618606 us; error 42 us; skew 500 ppm
I20240501 07:59:06.621039 8712 webserver.cc:460] Webserver started at http://127.0.99.133:37019/ using document root <none> and password file <none>
I20240501 07:59:06.621724 8712 fs_manager.cc:359] Metadata directory not provided
I20240501 07:59:06.621829 8712 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:59:06.625252 8712 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20240501 07:59:06.627213 8726 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:06.628439 8712 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 07:59:06.628548 8712 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "74077b61d6f54342bf9736b4a9f2eade"
format_stamp: "Formatted at 2024-05-01 07:58:50 on dist-test-slave-40g1"
I20240501 07:59:06.628991 8712 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:59:06.643509 8712 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:59:06.644120 8712 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 07:59:06.644287 8712 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:59:06.644822 8712 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 07:59:06.645745 8712 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 07:59:06.645794 8712 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:06.645860 8712 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 07:59:06.645887 8712 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:06.655447 8712 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.133:39859
I20240501 07:59:06.655457 8839 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.133:39859 every 8 connection(s)
I20240501 07:59:06.656646 8712 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 07:59:06.659966 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tbgEt/build/debug/bin/kudu as pid 8712
I20240501 07:59:06.660413 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tbgEt/build/debug/bin/kudu
/tmp/dist-test-task7tbgEt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.134:40141
--local_ip_for_outbound_sockets=127.0.99.134
--tserver_master_addrs=127.0.99.190:38383
--webserver_port=34395
--webserver_interface=127.0.99.134
--builtin_ntp_servers=127.0.99.148:34871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 07:59:06.667230 8840 heartbeater.cc:347] Connected to a master server at 127.0.99.190:38383
I20240501 07:59:06.667528 8840 heartbeater.cc:464] Registering TS with master...
I20240501 07:59:06.668313 8840 heartbeater.cc:511] Master 127.0.99.190:38383 requested a full tablet report, sending...
I20240501 07:59:06.669919 6168 ts_manager.cc:194] Re-registered known tserver with Master: 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133:39859)
I20240501 07:59:06.670709 6168 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.133:33663
W20240501 07:59:06.779290 8843 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:59:06.779515 8843 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 07:59:06.779559 8843 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:59:06.782673 8843 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 07:59:06.782744 8843 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 07:59:06.782835 8843 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.134
I20240501 07:59:06.786639 8843 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:34871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.134:40141
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.0.99.134
--webserver_port=34395
--tserver_master_addrs=127.0.99.190:38383
--never_fsync=true
--heap_profile_path=/tmp/kudu.8843
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.134
--log_dir=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 07:43:16 UTC on 1a15035a6830
build id 69
I20240501 07:59:06.787601 8843 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 07:59:06.788931 8843 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:59:06.798054 8849 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:59:06.798914 8848 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:59:06.799892 8851 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:59:06.800843 8843 server_base.cc:1034] running on GCE node
I20240501 07:59:06.801362 8843 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 07:59:06.802003 8843 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 07:59:06.803155 8843 hybrid_clock.cc:648] HybridClock initialized: now 1714550346803142 us; error 29 us; skew 500 ppm
I20240501 07:59:06.805249 8843 webserver.cc:460] Webserver started at http://127.0.99.134:34395/ using document root <none> and password file <none>
I20240501 07:59:06.805848 8843 fs_manager.cc:359] Metadata directory not provided
I20240501 07:59:06.805924 8843 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:59:06.809746 8843 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20240501 07:59:06.811906 8857 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:06.812947 8843 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 07:59:06.813094 8843 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "13d05ebdca8c4f489c597ce550dd773d"
format_stamp: "Formatted at 2024-05-01 07:58:50 on dist-test-slave-40g1"
I20240501 07:59:06.813508 8843 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:59:06.838538 8843 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:59:06.839191 8843 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 07:59:06.839349 8843 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:59:06.839918 8843 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 07:59:06.840997 8843 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 07:59:06.841059 8843 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:06.841111 8843 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 07:59:06.841127 8843 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:06.851558 8843 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.134:40141
I20240501 07:59:06.851588 8970 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.134:40141 every 8 connection(s)
I20240501 07:59:06.852629 8843 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240501 07:59:06.857004 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tbgEt/build/debug/bin/kudu as pid 8843
I20240501 07:59:06.857460 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tbgEt/build/debug/bin/kudu
/tmp/dist-test-task7tbgEt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.135:42607
--local_ip_for_outbound_sockets=127.0.99.135
--tserver_master_addrs=127.0.99.190:38383
--webserver_port=45825
--webserver_interface=127.0.99.135
--builtin_ntp_servers=127.0.99.148:34871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 07:59:06.863448 8971 heartbeater.cc:347] Connected to a master server at 127.0.99.190:38383
I20240501 07:59:06.863734 8971 heartbeater.cc:464] Registering TS with master...
I20240501 07:59:06.864463 8971 heartbeater.cc:511] Master 127.0.99.190:38383 requested a full tablet report, sending...
I20240501 07:59:06.865644 6168 ts_manager.cc:194] Re-registered known tserver with Master: 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134:40141)
I20240501 07:59:06.866456 6168 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.134:42147
W20240501 07:59:06.977283 8974 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:59:06.977522 8974 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 07:59:06.977567 8974 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:59:06.980710 8974 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 07:59:06.980790 8974 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 07:59:06.980861 8974 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.135
I20240501 07:59:06.984601 8974 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:34871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.135:42607
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.0.99.135
--webserver_port=45825
--tserver_master_addrs=127.0.99.190:38383
--never_fsync=true
--heap_profile_path=/tmp/kudu.8974
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.135
--log_dir=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 07:43:16 UTC on 1a15035a6830
build id 69
I20240501 07:59:06.985638 8974 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 07:59:06.986828 8974 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:59:06.996546 8980 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:59:06.997473 8979 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:59:06.998430 8974 server_base.cc:1034] running on GCE node
W20240501 07:59:06.999367 8982 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:59:06.999779 8974 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 07:59:07.000351 8974 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 07:59:07.001490 8974 hybrid_clock.cc:648] HybridClock initialized: now 1714550347001478 us; error 31 us; skew 500 ppm
I20240501 07:59:07.003499 8974 webserver.cc:460] Webserver started at http://127.0.99.135:45825/ using document root <none> and password file <none>
I20240501 07:59:07.004077 8974 fs_manager.cc:359] Metadata directory not provided
I20240501 07:59:07.004144 8974 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:59:07.008176 8974 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20240501 07:59:07.010462 8988 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:07.011561 8974 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240501 07:59:07.011657 8974 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "a505e7cb133c4392bfda8a4a20874ae9"
format_stamp: "Formatted at 2024-05-01 07:58:50 on dist-test-slave-40g1"
I20240501 07:59:07.012081 8974 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:59:07.028080 8974 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:59:07.028769 8974 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 07:59:07.028930 8974 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:59:07.029489 8974 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 07:59:07.030376 8974 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 07:59:07.030422 8974 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:07.030481 8974 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 07:59:07.030506 8974 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:07.039660 8974 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.135:42607
I20240501 07:59:07.039726 9101 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.135:42607 every 8 connection(s)
I20240501 07:59:07.040803 8974 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240501 07:59:07.043892 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tbgEt/build/debug/bin/kudu as pid 8974
I20240501 07:59:07.044276 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tbgEt/build/debug/bin/kudu
/tmp/dist-test-task7tbgEt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.136:46149
--local_ip_for_outbound_sockets=127.0.99.136
--tserver_master_addrs=127.0.99.190:38383
--webserver_port=41673
--webserver_interface=127.0.99.136
--builtin_ntp_servers=127.0.99.148:34871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 07:59:07.050760 9102 heartbeater.cc:347] Connected to a master server at 127.0.99.190:38383
I20240501 07:59:07.050999 9102 heartbeater.cc:464] Registering TS with master...
I20240501 07:59:07.051532 9102 heartbeater.cc:511] Master 127.0.99.190:38383 requested a full tablet report, sending...
I20240501 07:59:07.052465 6168 ts_manager.cc:194] Re-registered known tserver with Master: a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607)
I20240501 07:59:07.053042 6168 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.135:41229
W20240501 07:59:07.164149 9105 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:59:07.164403 9105 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 07:59:07.164469 9105 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:59:07.167690 9105 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 07:59:07.167778 9105 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 07:59:07.167878 9105 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.136
I20240501 07:59:07.171880 9105 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:34871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.136:46149
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.0.99.136
--webserver_port=41673
--tserver_master_addrs=127.0.99.190:38383
--never_fsync=true
--heap_profile_path=/tmp/kudu.9105
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.136
--log_dir=/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 07:43:16 UTC on 1a15035a6830
build id 69
I20240501 07:59:07.172884 9105 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 07:59:07.174090 9105 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 07:59:07.183115 9111 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:59:07.184034 9112 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 07:59:07.184862 9114 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 07:59:07.186002 9105 server_base.cc:1034] running on GCE node
I20240501 07:59:07.186702 9105 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 07:59:07.187409 9105 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 07:59:07.188584 9105 hybrid_clock.cc:648] HybridClock initialized: now 1714550347188569 us; error 33 us; skew 500 ppm
I20240501 07:59:07.190725 9105 webserver.cc:460] Webserver started at http://127.0.99.136:41673/ using document root <none> and password file <none>
I20240501 07:59:07.191398 9105 fs_manager.cc:359] Metadata directory not provided
I20240501 07:59:07.191525 9105 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 07:59:07.195377 9105 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20240501 07:59:07.197608 9121 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:07.198707 9105 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240501 07:59:07.198825 9105 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "83f20e84af4c40d09a4804b6b26e697f"
format_stamp: "Formatted at 2024-05-01 07:58:50 on dist-test-slave-40g1"
I20240501 07:59:07.199246 9105 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 07:59:07.220094 9105 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 07:59:07.220786 9105 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 07:59:07.220952 9105 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 07:59:07.221566 9105 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 07:59:07.222517 9105 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 07:59:07.222568 9105 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:07.222630 9105 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 07:59:07.222657 9105 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 07:59:07.232211 9105 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.136:46149
I20240501 07:59:07.232210 9234 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.136:46149 every 8 connection(s)
I20240501 07:59:07.233726 9105 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240501 07:59:07.240749 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tbgEt/build/debug/bin/kudu as pid 9105
I20240501 07:59:07.243077 9235 heartbeater.cc:347] Connected to a master server at 127.0.99.190:38383
I20240501 07:59:07.243360 9235 heartbeater.cc:464] Registering TS with master...
I20240501 07:59:07.244128 9235 heartbeater.cc:511] Master 127.0.99.190:38383 requested a full tablet report, sending...
I20240501 07:59:07.245159 6168 ts_manager.cc:194] Re-registered known tserver with Master: 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149)
I20240501 07:59:07.245723 6168 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.136:56391
I20240501 07:59:07.249207 6168 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55950:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 07:59:07.256024 6294 tablet_service.cc:1467] Processing CreateTablet for tablet 733ab980b9a442ed90b5a10657b8285e (DEFAULT_TABLE table=rebalancer_dynamic_table [id=0c9b6b6e953c40078226901a5da69a13]), partition=
I20240501 07:59:07.256342 6294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 733ab980b9a442ed90b5a10657b8285e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:07.259354 9251 tablet_bootstrap.cc:492] T 733ab980b9a442ed90b5a10657b8285e P 8b8b3ad3aa8f4053b2167dd550f2086f: Bootstrap starting.
I20240501 07:59:07.264009 9251 tablet_bootstrap.cc:654] T 733ab980b9a442ed90b5a10657b8285e P 8b8b3ad3aa8f4053b2167dd550f2086f: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:07.265152 9036 tablet_service.cc:1467] Processing CreateTablet for tablet 733ab980b9a442ed90b5a10657b8285e (DEFAULT_TABLE table=rebalancer_dynamic_table [id=0c9b6b6e953c40078226901a5da69a13]), partition=
I20240501 07:59:07.265873 9036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 733ab980b9a442ed90b5a10657b8285e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:07.266816 9251 tablet_bootstrap.cc:492] T 733ab980b9a442ed90b5a10657b8285e P 8b8b3ad3aa8f4053b2167dd550f2086f: No bootstrap required, opened a new log
I20240501 07:59:07.266919 9251 ts_tablet_manager.cc:1367] T 733ab980b9a442ed90b5a10657b8285e P 8b8b3ad3aa8f4053b2167dd550f2086f: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240501 07:59:07.267360 9251 raft_consensus.cc:348] T 733ab980b9a442ed90b5a10657b8285e P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } }
I20240501 07:59:07.267480 9251 raft_consensus.cc:374] T 733ab980b9a442ed90b5a10657b8285e P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:07.267509 9251 raft_consensus.cc:729] T 733ab980b9a442ed90b5a10657b8285e P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b8b3ad3aa8f4053b2167dd550f2086f, State: Initialized, Role: FOLLOWER
I20240501 07:59:07.267710 9251 consensus_queue.cc:260] T 733ab980b9a442ed90b5a10657b8285e P 8b8b3ad3aa8f4053b2167dd550f2086f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } }
I20240501 07:59:07.268072 9251 ts_tablet_manager.cc:1398] T 733ab980b9a442ed90b5a10657b8285e P 8b8b3ad3aa8f4053b2167dd550f2086f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:07.273010 9253 tablet_bootstrap.cc:492] T 733ab980b9a442ed90b5a10657b8285e P a505e7cb133c4392bfda8a4a20874ae9: Bootstrap starting.
I20240501 07:59:07.275400 9253 tablet_bootstrap.cc:654] T 733ab980b9a442ed90b5a10657b8285e P a505e7cb133c4392bfda8a4a20874ae9: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:07.276221 9253 log.cc:824] T 733ab980b9a442ed90b5a10657b8285e P a505e7cb133c4392bfda8a4a20874ae9: Log is configured to *not* fsync() on all Append() calls
I20240501 07:59:07.277055 9169 tablet_service.cc:1467] Processing CreateTablet for tablet 733ab980b9a442ed90b5a10657b8285e (DEFAULT_TABLE table=rebalancer_dynamic_table [id=0c9b6b6e953c40078226901a5da69a13]), partition=
I20240501 07:59:07.277797 9169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 733ab980b9a442ed90b5a10657b8285e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:07.278299 9253 tablet_bootstrap.cc:492] T 733ab980b9a442ed90b5a10657b8285e P a505e7cb133c4392bfda8a4a20874ae9: No bootstrap required, opened a new log
I20240501 07:59:07.278476 9253 ts_tablet_manager.cc:1367] T 733ab980b9a442ed90b5a10657b8285e P a505e7cb133c4392bfda8a4a20874ae9: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20240501 07:59:07.282526 9253 raft_consensus.cc:348] T 733ab980b9a442ed90b5a10657b8285e P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } }
I20240501 07:59:07.282740 9253 raft_consensus.cc:374] T 733ab980b9a442ed90b5a10657b8285e P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:07.282811 9253 raft_consensus.cc:729] T 733ab980b9a442ed90b5a10657b8285e P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a505e7cb133c4392bfda8a4a20874ae9, State: Initialized, Role: FOLLOWER
I20240501 07:59:07.283244 9253 consensus_queue.cc:260] T 733ab980b9a442ed90b5a10657b8285e P a505e7cb133c4392bfda8a4a20874ae9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } }
I20240501 07:59:07.283943 9102 heartbeater.cc:503] Master 127.0.99.190:38383 was elected leader, sending a full tablet report...
I20240501 07:59:07.284219 9253 ts_tablet_manager.cc:1398] T 733ab980b9a442ed90b5a10657b8285e P a505e7cb133c4392bfda8a4a20874ae9: Time spent starting tablet: real 0.006s user 0.000s sys 0.005s
I20240501 07:59:07.287153 9256 tablet_bootstrap.cc:492] T 733ab980b9a442ed90b5a10657b8285e P 83f20e84af4c40d09a4804b6b26e697f: Bootstrap starting.
I20240501 07:59:07.290087 9256 tablet_bootstrap.cc:654] T 733ab980b9a442ed90b5a10657b8285e P 83f20e84af4c40d09a4804b6b26e697f: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:07.290940 9256 log.cc:824] T 733ab980b9a442ed90b5a10657b8285e P 83f20e84af4c40d09a4804b6b26e697f: Log is configured to *not* fsync() on all Append() calls
W20240501 07:59:07.291767 9103 tablet.cc:2369] T 733ab980b9a442ed90b5a10657b8285e P a505e7cb133c4392bfda8a4a20874ae9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 07:59:07.292793 9256 tablet_bootstrap.cc:492] T 733ab980b9a442ed90b5a10657b8285e P 83f20e84af4c40d09a4804b6b26e697f: No bootstrap required, opened a new log
I20240501 07:59:07.293040 9256 ts_tablet_manager.cc:1367] T 733ab980b9a442ed90b5a10657b8285e P 83f20e84af4c40d09a4804b6b26e697f: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20240501 07:59:07.297469 9256 raft_consensus.cc:348] T 733ab980b9a442ed90b5a10657b8285e P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } }
I20240501 07:59:07.297755 9256 raft_consensus.cc:374] T 733ab980b9a442ed90b5a10657b8285e P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:07.297890 9256 raft_consensus.cc:729] T 733ab980b9a442ed90b5a10657b8285e P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83f20e84af4c40d09a4804b6b26e697f, State: Initialized, Role: FOLLOWER
I20240501 07:59:07.298377 9256 consensus_queue.cc:260] T 733ab980b9a442ed90b5a10657b8285e P 83f20e84af4c40d09a4804b6b26e697f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } }
I20240501 07:59:07.299099 9235 heartbeater.cc:503] Master 127.0.99.190:38383 was elected leader, sending a full tablet report...
I20240501 07:59:07.300513 9256 ts_tablet_manager.cc:1398] T 733ab980b9a442ed90b5a10657b8285e P 83f20e84af4c40d09a4804b6b26e697f: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20240501 07:59:07.420437 8447 raft_consensus.cc:482] T 733ab980b9a442ed90b5a10657b8285e P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:07.420588 8447 raft_consensus.cc:504] T 733ab980b9a442ed90b5a10657b8285e P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } }
I20240501 07:59:07.421099 8447 leader_election.cc:288] T 733ab980b9a442ed90b5a10657b8285e P 8b8b3ad3aa8f4053b2167dd550f2086f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607), 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149)
I20240501 07:59:07.426395 9056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "733ab980b9a442ed90b5a10657b8285e" candidate_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a505e7cb133c4392bfda8a4a20874ae9" is_pre_election: true
I20240501 07:59:07.426784 9056 raft_consensus.cc:2454] T 733ab980b9a442ed90b5a10657b8285e P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b8b3ad3aa8f4053b2167dd550f2086f in term 0.
I20240501 07:59:07.427254 6249 leader_election.cc:302] T 733ab980b9a442ed90b5a10657b8285e P 8b8b3ad3aa8f4053b2167dd550f2086f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b8b3ad3aa8f4053b2167dd550f2086f, a505e7cb133c4392bfda8a4a20874ae9; no voters:
I20240501 07:59:07.427469 8447 raft_consensus.cc:2781] T 733ab980b9a442ed90b5a10657b8285e P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:07.427526 8447 raft_consensus.cc:482] T 733ab980b9a442ed90b5a10657b8285e P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:07.427575 8447 raft_consensus.cc:3037] T 733ab980b9a442ed90b5a10657b8285e P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:07.428222 9189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "733ab980b9a442ed90b5a10657b8285e" candidate_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83f20e84af4c40d09a4804b6b26e697f" is_pre_election: true
I20240501 07:59:07.428471 9189 raft_consensus.cc:2454] T 733ab980b9a442ed90b5a10657b8285e P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b8b3ad3aa8f4053b2167dd550f2086f in term 0.
I20240501 07:59:07.428524 8447 raft_consensus.cc:504] T 733ab980b9a442ed90b5a10657b8285e P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } }
I20240501 07:59:07.428869 8447 leader_election.cc:288] T 733ab980b9a442ed90b5a10657b8285e P 8b8b3ad3aa8f4053b2167dd550f2086f [CANDIDATE]: Term 1 election: Requested vote from peers a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607), 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149)
I20240501 07:59:07.429302 9189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "733ab980b9a442ed90b5a10657b8285e" candidate_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83f20e84af4c40d09a4804b6b26e697f"
I20240501 07:59:07.429414 9189 raft_consensus.cc:3037] T 733ab980b9a442ed90b5a10657b8285e P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:07.430534 9189 raft_consensus.cc:2454] T 733ab980b9a442ed90b5a10657b8285e P 83f20e84af4c40d09a4804b6b26e697f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b8b3ad3aa8f4053b2167dd550f2086f in term 1.
I20240501 07:59:07.430781 9056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "733ab980b9a442ed90b5a10657b8285e" candidate_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a505e7cb133c4392bfda8a4a20874ae9"
I20240501 07:59:07.430819 6249 leader_election.cc:302] T 733ab980b9a442ed90b5a10657b8285e P 8b8b3ad3aa8f4053b2167dd550f2086f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83f20e84af4c40d09a4804b6b26e697f, 8b8b3ad3aa8f4053b2167dd550f2086f; no voters:
I20240501 07:59:07.430914 9056 raft_consensus.cc:3037] T 733ab980b9a442ed90b5a10657b8285e P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:07.430986 8447 raft_consensus.cc:2781] T 733ab980b9a442ed90b5a10657b8285e P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:07.431064 8447 raft_consensus.cc:686] T 733ab980b9a442ed90b5a10657b8285e P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 LEADER]: Becoming Leader. State: Replica: 8b8b3ad3aa8f4053b2167dd550f2086f, State: Running, Role: LEADER
I20240501 07:59:07.431211 8447 consensus_queue.cc:237] T 733ab980b9a442ed90b5a10657b8285e P 8b8b3ad3aa8f4053b2167dd550f2086f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } }
I20240501 07:59:07.432121 9056 raft_consensus.cc:2454] T 733ab980b9a442ed90b5a10657b8285e P a505e7cb133c4392bfda8a4a20874ae9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b8b3ad3aa8f4053b2167dd550f2086f in term 1.
I20240501 07:59:07.434031 6168 catalog_manager.cc:5484] T 733ab980b9a442ed90b5a10657b8285e P 8b8b3ad3aa8f4053b2167dd550f2086f reported cstate change: term changed from 0 to 1, leader changed from <none> to 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129). New cstate: current_term: 1 leader_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } health_report { overall_health: HEALTHY } } }
W20240501 07:59:07.484771 9236 tablet.cc:2369] T 733ab980b9a442ed90b5a10657b8285e P 83f20e84af4c40d09a4804b6b26e697f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 07:59:07.517045 8905 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:07.517046 8757 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:07.518661 6556 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 07:59:07.522204 9036 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:07.525254 9169 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:07.548589 6425 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 07:59:07.551227 6172 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55950:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 07:59:07.560278 9056 raft_consensus.cc:1261] T 733ab980b9a442ed90b5a10657b8285e P a505e7cb133c4392bfda8a4a20874ae9 [term 1 FOLLOWER]: Refusing update from remote peer 8b8b3ad3aa8f4053b2167dd550f2086f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:07.560355 9189 raft_consensus.cc:1261] T 733ab980b9a442ed90b5a10657b8285e P 83f20e84af4c40d09a4804b6b26e697f [term 1 FOLLOWER]: Refusing update from remote peer 8b8b3ad3aa8f4053b2167dd550f2086f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:07.561007 9106 consensus_queue.cc:1035] T 733ab980b9a442ed90b5a10657b8285e P 8b8b3ad3aa8f4053b2167dd550f2086f [LEADER]: Connected to new peer: Peer: permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:07.564167 9106 consensus_queue.cc:1035] T 733ab980b9a442ed90b5a10657b8285e P 8b8b3ad3aa8f4053b2167dd550f2086f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:07.565999 6172 catalog_manager.cc:5576] TS 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149) does not have the latest schema for tablet 733ab980b9a442ed90b5a10657b8285e (table rebalancer_dynamic_table [id=0c9b6b6e953c40078226901a5da69a13]). Expected version 1 got 0
I20240501 07:59:07.569046 6294 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 07:59:07.569802 9323 tablet.cc:1722] T 733ab980b9a442ed90b5a10657b8285e P 83f20e84af4c40d09a4804b6b26e697f: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 07:59:07.568552 9322 tablet.cc:1722] T 733ab980b9a442ed90b5a10657b8285e P 8b8b3ad3aa8f4053b2167dd550f2086f: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 07:59:07.570734 9323 tablet.cc:1620] T 733ab980b9a442ed90b5a10657b8285e P 83f20e84af4c40d09a4804b6b26e697f: MemRowSet was empty: no flush needed.
I20240501 07:59:07.570780 9322 tablet.cc:1620] T 733ab980b9a442ed90b5a10657b8285e P 8b8b3ad3aa8f4053b2167dd550f2086f: MemRowSet was empty: no flush needed.
I20240501 07:59:07.573500 6164 catalog_manager.cc:5576] TS a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607) does not have the latest schema for tablet 733ab980b9a442ed90b5a10657b8285e (table rebalancer_dynamic_table [id=0c9b6b6e953c40078226901a5da69a13]). Expected version 1 got 0
I20240501 07:59:07.573885 6686 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 07:59:07.575158 6172 catalog_manager.cc:5576] TS 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129:46875) does not have the latest schema for tablet 733ab980b9a442ed90b5a10657b8285e (table rebalancer_dynamic_table [id=0c9b6b6e953c40078226901a5da69a13]). Expected version 1 got 0
I20240501 07:59:07.576614 6169 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 0bc706e85ce74cef97a4827ea4d4aa65: rebalancer_dynamic_table [id=0c9b6b6e953c40078226901a5da69a13] alter complete (version 1)
I20240501 07:59:07.585908 6164 catalog_manager.cc:5576] TS a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607) does not have the latest schema for tablet 733ab980b9a442ed90b5a10657b8285e (table rebalancer_dynamic_table [id=0c9b6b6e953c40078226901a5da69a13]). Expected version 1 got 0
I20240501 07:59:07.586275 9326 tablet.cc:1722] T 733ab980b9a442ed90b5a10657b8285e P a505e7cb133c4392bfda8a4a20874ae9: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 07:59:07.587095 9326 tablet.cc:1620] T 733ab980b9a442ed90b5a10657b8285e P a505e7cb133c4392bfda8a4a20874ae9: MemRowSet was empty: no flush needed.
I20240501 07:59:07.593055 9324 mvcc.cc:205] Tried to move back new op lower bound from 7022798223600615424 to 7022798223079890944. Current Snapshot: MvccSnapshot[applied={T|T < 7022798223600615424}]
I20240501 07:59:07.673599 8840 heartbeater.cc:503] Master 127.0.99.190:38383 was elected leader, sending a full tablet report...
I20240501 07:59:07.689266 6168 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55950:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 07:59:07.698441 9322 tablet.cc:1722] T 733ab980b9a442ed90b5a10657b8285e P 8b8b3ad3aa8f4053b2167dd550f2086f: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 07:59:07.698616 9326 tablet.cc:1722] T 733ab980b9a442ed90b5a10657b8285e P a505e7cb133c4392bfda8a4a20874ae9: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 07:59:07.698817 9322 tablet.cc:1620] T 733ab980b9a442ed90b5a10657b8285e P 8b8b3ad3aa8f4053b2167dd550f2086f: MemRowSet was empty: no flush needed.
I20240501 07:59:07.698923 9326 tablet.cc:1620] T 733ab980b9a442ed90b5a10657b8285e P a505e7cb133c4392bfda8a4a20874ae9: MemRowSet was empty: no flush needed.
I20240501 07:59:07.699188 9323 tablet.cc:1722] T 733ab980b9a442ed90b5a10657b8285e P 83f20e84af4c40d09a4804b6b26e697f: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 07:59:07.699484 9323 tablet.cc:1620] T 733ab980b9a442ed90b5a10657b8285e P 83f20e84af4c40d09a4804b6b26e697f: MemRowSet was empty: no flush needed.
I20240501 07:59:07.704897 6169 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 0bc706e85ce74cef97a4827ea4d4aa65: rebalancer_dynamic_table [id=0c9b6b6e953c40078226901a5da69a13] alter complete (version 2)
I20240501 07:59:07.730161 8905 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:07.730412 8757 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:07.737977 9036 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:07.741180 6556 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 07:59:07.741777 6425 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 07:59:07.742533 6166 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55950:
table { table_name: "rebalance_test_table_0" } modify_external_catalogs: true
I20240501 07:59:07.742692 6166 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55950:
table { table_name: "rebalance_test_table_0" } modify_external_catalogs: true
I20240501 07:59:07.743804 6171 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55950:
name: "rebalancer_extra_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 07:59:07.745748 6294 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 07:59:07.746702 6166 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 0bc706e85ce74cef97a4827ea4d4aa65: Sending DeleteTablet for 3 replicas of tablet def178155f14416391e89ba972f4d723
I20240501 07:59:07.747115 6166 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 0bc706e85ce74cef97a4827ea4d4aa65: Sending DeleteTablet for 3 replicas of tablet 8480c7b51a914f03bcad7701d75e4dd4
I20240501 07:59:07.747366 6166 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 0bc706e85ce74cef97a4827ea4d4aa65: Sending DeleteTablet for 3 replicas of tablet e04fe36b8efa4f9584e093279f7addf4
I20240501 07:59:07.747469 6556 tablet_service.cc:1514] Processing DeleteTablet for tablet def178155f14416391e89ba972f4d723 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:07 UTC) from {username='slave'} at 127.0.0.1:53874
I20240501 07:59:07.747999 6425 tablet_service.cc:1514] Processing DeleteTablet for tablet def178155f14416391e89ba972f4d723 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:07 UTC) from {username='slave'} at 127.0.0.1:45034
I20240501 07:59:07.748327 6424 tablet_service.cc:1514] Processing DeleteTablet for tablet e04fe36b8efa4f9584e093279f7addf4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:07 UTC) from {username='slave'} at 127.0.0.1:45034
I20240501 07:59:07.748030 6426 tablet_service.cc:1514] Processing DeleteTablet for tablet 8480c7b51a914f03bcad7701d75e4dd4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:07 UTC) from {username='slave'} at 127.0.0.1:45034
I20240501 07:59:07.748543 6556 tablet_service.cc:1514] Processing DeleteTablet for tablet 8480c7b51a914f03bcad7701d75e4dd4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:07 UTC) from {username='slave'} at 127.0.0.1:53874
I20240501 07:59:07.751610 6686 tablet_service.cc:1514] Processing DeleteTablet for tablet def178155f14416391e89ba972f4d723 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:07 UTC) from {username='slave'} at 127.0.0.1:38524
I20240501 07:59:07.751974 6687 tablet_service.cc:1514] Processing DeleteTablet for tablet e04fe36b8efa4f9584e093279f7addf4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:07 UTC) from {username='slave'} at 127.0.0.1:38524
I20240501 07:59:07.752148 6295 tablet_service.cc:1514] Processing DeleteTablet for tablet e04fe36b8efa4f9584e093279f7addf4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:07 UTC) from {username='slave'} at 127.0.0.1:45480
I20240501 07:59:07.752498 6294 tablet_service.cc:1514] Processing DeleteTablet for tablet 8480c7b51a914f03bcad7701d75e4dd4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:07 UTC) from {username='slave'} at 127.0.0.1:45480
I20240501 07:59:07.753214 9388 tablet_replica.cc:330] T e04fe36b8efa4f9584e093279f7addf4 P 8b8b3ad3aa8f4053b2167dd550f2086f: stopping tablet replica
I20240501 07:59:07.753594 9388 raft_consensus.cc:2229] T e04fe36b8efa4f9584e093279f7addf4 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 LEADER]: Raft consensus shutting down.
I20240501 07:59:07.754010 9388 raft_consensus.cc:2258] T e04fe36b8efa4f9584e093279f7addf4 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:07.755162 9388 ts_tablet_manager.cc:1875] T e04fe36b8efa4f9584e093279f7addf4 P 8b8b3ad3aa8f4053b2167dd550f2086f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:59:07.757797 6556 tablet_service.cc:1467] Processing CreateTablet for tablet 6292ff0c7e624dd7b24715b83a30fff8 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=9ec2540a0f4b49ada014e0c68ec3b2a0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:07.758086 6556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6292ff0c7e624dd7b24715b83a30fff8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:07.759145 9388 ts_tablet_manager.cc:1888] T e04fe36b8efa4f9584e093279f7addf4 P 8b8b3ad3aa8f4053b2167dd550f2086f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 07:59:07.759289 9388 log.cc:1196] T e04fe36b8efa4f9584e093279f7addf4 P 8b8b3ad3aa8f4053b2167dd550f2086f: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/e04fe36b8efa4f9584e093279f7addf4
I20240501 07:59:07.760533 9036 tablet_service.cc:1467] Processing CreateTablet for tablet f88fe033bce14bf0b1b63a52a7ba137e (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=9ec2540a0f4b49ada014e0c68ec3b2a0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:07.760804 9169 tablet_service.cc:1467] Processing CreateTablet for tablet 6292ff0c7e624dd7b24715b83a30fff8 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=9ec2540a0f4b49ada014e0c68ec3b2a0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:07.760929 9036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f88fe033bce14bf0b1b63a52a7ba137e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:07.761154 9169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6292ff0c7e624dd7b24715b83a30fff8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:07.763258 9253 tablet_bootstrap.cc:492] T f88fe033bce14bf0b1b63a52a7ba137e P a505e7cb133c4392bfda8a4a20874ae9: Bootstrap starting.
I20240501 07:59:07.763504 9168 tablet_service.cc:1467] Processing CreateTablet for tablet f88fe033bce14bf0b1b63a52a7ba137e (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=9ec2540a0f4b49ada014e0c68ec3b2a0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:07.763849 9168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f88fe033bce14bf0b1b63a52a7ba137e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:07.764513 9253 tablet_bootstrap.cc:654] T f88fe033bce14bf0b1b63a52a7ba137e P a505e7cb133c4392bfda8a4a20874ae9: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:07.765233 9167 tablet_service.cc:1467] Processing CreateTablet for tablet fb6b46072a8d46e18b53b9f8de702859 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=9ec2540a0f4b49ada014e0c68ec3b2a0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:07.765381 6687 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 07:59:07.765451 9167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb6b46072a8d46e18b53b9f8de702859. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:07.766489 9256 tablet_bootstrap.cc:492] T 6292ff0c7e624dd7b24715b83a30fff8 P 83f20e84af4c40d09a4804b6b26e697f: Bootstrap starting.
I20240501 07:59:07.767336 9256 tablet_bootstrap.cc:654] T 6292ff0c7e624dd7b24715b83a30fff8 P 83f20e84af4c40d09a4804b6b26e697f: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:07.771448 9169 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:07.772646 8757 tablet_service.cc:1467] Processing CreateTablet for tablet 6292ff0c7e624dd7b24715b83a30fff8 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=9ec2540a0f4b49ada014e0c68ec3b2a0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:07.773146 8905 tablet_service.cc:1467] Processing CreateTablet for tablet f88fe033bce14bf0b1b63a52a7ba137e (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=9ec2540a0f4b49ada014e0c68ec3b2a0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:07.773306 9388 ts_tablet_manager.cc:1909] T e04fe36b8efa4f9584e093279f7addf4 P 8b8b3ad3aa8f4053b2167dd550f2086f: Deleting consensus metadata
I20240501 07:59:07.773674 8757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6292ff0c7e624dd7b24715b83a30fff8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:07.773700 8905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f88fe033bce14bf0b1b63a52a7ba137e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:07.775079 6157 catalog_manager.cc:4830] TS 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129:46875): tablet e04fe36b8efa4f9584e093279f7addf4 (table rebalance_test_table_0 [id=6448eafcd04b41cba44dc175507acfc0]) successfully deleted
I20240501 07:59:07.775319 8774 tablet_service.cc:1467] Processing CreateTablet for tablet fb6b46072a8d46e18b53b9f8de702859 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=9ec2540a0f4b49ada014e0c68ec3b2a0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:07.775545 8774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb6b46072a8d46e18b53b9f8de702859. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:07.772642 8904 tablet_service.cc:1467] Processing CreateTablet for tablet fb6b46072a8d46e18b53b9f8de702859 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=9ec2540a0f4b49ada014e0c68ec3b2a0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:07.783605 8904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb6b46072a8d46e18b53b9f8de702859. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:07.788830 9388 tablet_replica.cc:330] T 8480c7b51a914f03bcad7701d75e4dd4 P 8b8b3ad3aa8f4053b2167dd550f2086f: stopping tablet replica
I20240501 07:59:07.789098 9388 raft_consensus.cc:2229] T 8480c7b51a914f03bcad7701d75e4dd4 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:07.789288 9388 raft_consensus.cc:2258] T 8480c7b51a914f03bcad7701d75e4dd4 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:07.790316 9388 ts_tablet_manager.cc:1875] T 8480c7b51a914f03bcad7701d75e4dd4 P 8b8b3ad3aa8f4053b2167dd550f2086f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:59:07.791002 9387 tablet_replica.cc:330] T def178155f14416391e89ba972f4d723 P 1f9cb72be98941b5a8e504cbe34132b9: stopping tablet replica
I20240501 07:59:07.791323 9387 raft_consensus.cc:2229] T def178155f14416391e89ba972f4d723 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:07.791515 9387 raft_consensus.cc:2258] T def178155f14416391e89ba972f4d723 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:07.792735 9253 tablet_bootstrap.cc:492] T f88fe033bce14bf0b1b63a52a7ba137e P a505e7cb133c4392bfda8a4a20874ae9: No bootstrap required, opened a new log
I20240501 07:59:07.792822 9253 ts_tablet_manager.cc:1367] T f88fe033bce14bf0b1b63a52a7ba137e P a505e7cb133c4392bfda8a4a20874ae9: Time spent bootstrapping tablet: real 0.030s user 0.002s sys 0.000s
I20240501 07:59:07.793422 9253 raft_consensus.cc:348] T f88fe033bce14bf0b1b63a52a7ba137e P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } }
I20240501 07:59:07.793531 9253 raft_consensus.cc:374] T f88fe033bce14bf0b1b63a52a7ba137e P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:07.793560 9253 raft_consensus.cc:729] T f88fe033bce14bf0b1b63a52a7ba137e P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a505e7cb133c4392bfda8a4a20874ae9, State: Initialized, Role: FOLLOWER
I20240501 07:59:07.793687 9253 consensus_queue.cc:260] T f88fe033bce14bf0b1b63a52a7ba137e P a505e7cb133c4392bfda8a4a20874ae9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } }
I20240501 07:59:07.793922 9253 ts_tablet_manager.cc:1398] T f88fe033bce14bf0b1b63a52a7ba137e P a505e7cb133c4392bfda8a4a20874ae9: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 07:59:07.795501 9256 tablet_bootstrap.cc:492] T 6292ff0c7e624dd7b24715b83a30fff8 P 83f20e84af4c40d09a4804b6b26e697f: No bootstrap required, opened a new log
I20240501 07:59:07.795588 9256 ts_tablet_manager.cc:1367] T 6292ff0c7e624dd7b24715b83a30fff8 P 83f20e84af4c40d09a4804b6b26e697f: Time spent bootstrapping tablet: real 0.029s user 0.002s sys 0.000s
I20240501 07:59:07.795877 9387 ts_tablet_manager.cc:1875] T def178155f14416391e89ba972f4d723 P 1f9cb72be98941b5a8e504cbe34132b9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:59:07.796123 9256 raft_consensus.cc:348] T 6292ff0c7e624dd7b24715b83a30fff8 P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } }
I20240501 07:59:07.796231 9256 raft_consensus.cc:374] T 6292ff0c7e624dd7b24715b83a30fff8 P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:07.796264 9256 raft_consensus.cc:729] T 6292ff0c7e624dd7b24715b83a30fff8 P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83f20e84af4c40d09a4804b6b26e697f, State: Initialized, Role: FOLLOWER
I20240501 07:59:07.796387 9256 consensus_queue.cc:260] T 6292ff0c7e624dd7b24715b83a30fff8 P 83f20e84af4c40d09a4804b6b26e697f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } }
I20240501 07:59:07.796640 9256 ts_tablet_manager.cc:1398] T 6292ff0c7e624dd7b24715b83a30fff8 P 83f20e84af4c40d09a4804b6b26e697f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:07.796803 9256 tablet_bootstrap.cc:492] T f88fe033bce14bf0b1b63a52a7ba137e P 83f20e84af4c40d09a4804b6b26e697f: Bootstrap starting.
I20240501 07:59:07.798158 9256 tablet_bootstrap.cc:654] T f88fe033bce14bf0b1b63a52a7ba137e P 83f20e84af4c40d09a4804b6b26e697f: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:07.798787 9387 ts_tablet_manager.cc:1888] T def178155f14416391e89ba972f4d723 P 1f9cb72be98941b5a8e504cbe34132b9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 07:59:07.798866 9387 log.cc:1196] T def178155f14416391e89ba972f4d723 P 1f9cb72be98941b5a8e504cbe34132b9: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/def178155f14416391e89ba972f4d723
I20240501 07:59:07.799170 9387 ts_tablet_manager.cc:1909] T def178155f14416391e89ba972f4d723 P 1f9cb72be98941b5a8e504cbe34132b9: Deleting consensus metadata
I20240501 07:59:07.800714 9387 tablet_replica.cc:330] T e04fe36b8efa4f9584e093279f7addf4 P 1f9cb72be98941b5a8e504cbe34132b9: stopping tablet replica
I20240501 07:59:07.800756 6159 catalog_manager.cc:4830] TS 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132:36773): tablet def178155f14416391e89ba972f4d723 (table rebalance_test_table_0 [id=6448eafcd04b41cba44dc175507acfc0]) successfully deleted
I20240501 07:59:07.800856 9387 raft_consensus.cc:2229] T e04fe36b8efa4f9584e093279f7addf4 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:07.801008 9387 raft_consensus.cc:2258] T e04fe36b8efa4f9584e093279f7addf4 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:07.801883 9387 ts_tablet_manager.cc:1875] T e04fe36b8efa4f9584e093279f7addf4 P 1f9cb72be98941b5a8e504cbe34132b9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:59:07.804266 9387 ts_tablet_manager.cc:1888] T e04fe36b8efa4f9584e093279f7addf4 P 1f9cb72be98941b5a8e504cbe34132b9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 07:59:07.804235 9385 tablet_replica.cc:330] T def178155f14416391e89ba972f4d723 P 3b164cbbc00f4e8eb3254ddd349a3ff9: stopping tablet replica
I20240501 07:59:07.804378 9387 log.cc:1196] T e04fe36b8efa4f9584e093279f7addf4 P 1f9cb72be98941b5a8e504cbe34132b9: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/e04fe36b8efa4f9584e093279f7addf4
I20240501 07:59:07.804700 9385 raft_consensus.cc:2229] T def178155f14416391e89ba972f4d723 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 LEADER]: Raft consensus shutting down.
I20240501 07:59:07.805107 9385 raft_consensus.cc:2258] T def178155f14416391e89ba972f4d723 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:07.807644 9393 tablet_bootstrap.cc:492] T fb6b46072a8d46e18b53b9f8de702859 P 13d05ebdca8c4f489c597ce550dd773d: Bootstrap starting.
I20240501 07:59:07.808533 9386 tablet_replica.cc:330] T def178155f14416391e89ba972f4d723 P 8315d22c56f7439a9e1fc82f153f018c: stopping tablet replica
I20240501 07:59:07.808825 9392 tablet_bootstrap.cc:492] T fb6b46072a8d46e18b53b9f8de702859 P 74077b61d6f54342bf9736b4a9f2eade: Bootstrap starting.
I20240501 07:59:07.811117 6171 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55950:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 07:59:07.811342 9386 raft_consensus.cc:2229] T def178155f14416391e89ba972f4d723 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:07.811545 9386 raft_consensus.cc:2258] T def178155f14416391e89ba972f4d723 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:07.812630 9386 ts_tablet_manager.cc:1875] T def178155f14416391e89ba972f4d723 P 8315d22c56f7439a9e1fc82f153f018c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:59:07.813262 9387 ts_tablet_manager.cc:1909] T e04fe36b8efa4f9584e093279f7addf4 P 1f9cb72be98941b5a8e504cbe34132b9: Deleting consensus metadata
I20240501 07:59:07.815141 6159 catalog_manager.cc:4830] TS 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132:36773): tablet e04fe36b8efa4f9584e093279f7addf4 (table rebalance_test_table_0 [id=6448eafcd04b41cba44dc175507acfc0]) successfully deleted
I20240501 07:59:07.815999 9392 tablet_bootstrap.cc:654] T fb6b46072a8d46e18b53b9f8de702859 P 74077b61d6f54342bf9736b4a9f2eade: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:07.818306 9393 tablet_bootstrap.cc:654] T fb6b46072a8d46e18b53b9f8de702859 P 13d05ebdca8c4f489c597ce550dd773d: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:07.807297 9391 tablet_bootstrap.cc:492] T 6292ff0c7e624dd7b24715b83a30fff8 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Bootstrap starting.
I20240501 07:59:07.822008 9391 tablet_bootstrap.cc:654] T 6292ff0c7e624dd7b24715b83a30fff8 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:07.822348 9385 ts_tablet_manager.cc:1875] T def178155f14416391e89ba972f4d723 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:59:07.823246 9322 tablet.cc:1722] T 733ab980b9a442ed90b5a10657b8285e P 8b8b3ad3aa8f4053b2167dd550f2086f: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 07:59:07.824817 9326 tablet.cc:1722] T 733ab980b9a442ed90b5a10657b8285e P a505e7cb133c4392bfda8a4a20874ae9: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 07:59:07.826181 9392 log.cc:824] T fb6b46072a8d46e18b53b9f8de702859 P 74077b61d6f54342bf9736b4a9f2eade: Log is configured to *not* fsync() on all Append() calls
I20240501 07:59:07.820909 9393 log.cc:824] T fb6b46072a8d46e18b53b9f8de702859 P 13d05ebdca8c4f489c597ce550dd773d: Log is configured to *not* fsync() on all Append() calls
I20240501 07:59:07.828245 9323 tablet.cc:1722] T 733ab980b9a442ed90b5a10657b8285e P 83f20e84af4c40d09a4804b6b26e697f: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 07:59:07.828302 9385 ts_tablet_manager.cc:1888] T def178155f14416391e89ba972f4d723 P 3b164cbbc00f4e8eb3254ddd349a3ff9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 07:59:07.828395 9385 log.cc:1196] T def178155f14416391e89ba972f4d723 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/def178155f14416391e89ba972f4d723
I20240501 07:59:07.828732 9385 ts_tablet_manager.cc:1909] T def178155f14416391e89ba972f4d723 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Deleting consensus metadata
I20240501 07:59:07.829420 9386 ts_tablet_manager.cc:1888] T def178155f14416391e89ba972f4d723 P 8315d22c56f7439a9e1fc82f153f018c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 07:59:07.831431 9386 log.cc:1196] T def178155f14416391e89ba972f4d723 P 8315d22c56f7439a9e1fc82f153f018c: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/def178155f14416391e89ba972f4d723
I20240501 07:59:07.831851 9386 ts_tablet_manager.cc:1909] T def178155f14416391e89ba972f4d723 P 8315d22c56f7439a9e1fc82f153f018c: Deleting consensus metadata
I20240501 07:59:07.832219 6160 catalog_manager.cc:4830] TS 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131:34471): tablet def178155f14416391e89ba972f4d723 (table rebalance_test_table_0 [id=6448eafcd04b41cba44dc175507acfc0]) successfully deleted
I20240501 07:59:07.832311 9391 tablet_bootstrap.cc:492] T 6292ff0c7e624dd7b24715b83a30fff8 P 3b164cbbc00f4e8eb3254ddd349a3ff9: No bootstrap required, opened a new log
I20240501 07:59:07.832414 9391 ts_tablet_manager.cc:1367] T 6292ff0c7e624dd7b24715b83a30fff8 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Time spent bootstrapping tablet: real 0.025s user 0.001s sys 0.000s
I20240501 07:59:07.833055 6159 catalog_manager.cc:4830] TS 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130:32941): tablet def178155f14416391e89ba972f4d723 (table rebalance_test_table_0 [id=6448eafcd04b41cba44dc175507acfc0]) successfully deleted
I20240501 07:59:07.833148 9391 raft_consensus.cc:348] T 6292ff0c7e624dd7b24715b83a30fff8 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } }
I20240501 07:59:07.833317 9391 raft_consensus.cc:374] T 6292ff0c7e624dd7b24715b83a30fff8 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:07.833432 9391 raft_consensus.cc:729] T 6292ff0c7e624dd7b24715b83a30fff8 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b164cbbc00f4e8eb3254ddd349a3ff9, State: Initialized, Role: FOLLOWER
I20240501 07:59:07.833600 9391 consensus_queue.cc:260] T 6292ff0c7e624dd7b24715b83a30fff8 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } }
I20240501 07:59:07.833969 9391 ts_tablet_manager.cc:1398] T 6292ff0c7e624dd7b24715b83a30fff8 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:07.834190 9385 tablet_replica.cc:330] T 8480c7b51a914f03bcad7701d75e4dd4 P 3b164cbbc00f4e8eb3254ddd349a3ff9: stopping tablet replica
I20240501 07:59:07.834331 9385 raft_consensus.cc:2229] T 8480c7b51a914f03bcad7701d75e4dd4 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 LEADER]: Raft consensus shutting down.
I20240501 07:59:07.834607 9385 raft_consensus.cc:2258] T 8480c7b51a914f03bcad7701d75e4dd4 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:07.835202 9385 ts_tablet_manager.cc:1875] T 8480c7b51a914f03bcad7701d75e4dd4 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:59:07.835518 9386 tablet_replica.cc:330] T e04fe36b8efa4f9584e093279f7addf4 P 8315d22c56f7439a9e1fc82f153f018c: stopping tablet replica
I20240501 07:59:07.835664 9386 raft_consensus.cc:2229] T e04fe36b8efa4f9584e093279f7addf4 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:07.835795 9386 raft_consensus.cc:2258] T e04fe36b8efa4f9584e093279f7addf4 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:07.836334 9386 ts_tablet_manager.cc:1875] T e04fe36b8efa4f9584e093279f7addf4 P 8315d22c56f7439a9e1fc82f153f018c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:59:07.838004 9385 ts_tablet_manager.cc:1888] T 8480c7b51a914f03bcad7701d75e4dd4 P 3b164cbbc00f4e8eb3254ddd349a3ff9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 07:59:07.838081 9385 log.cc:1196] T 8480c7b51a914f03bcad7701d75e4dd4 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/8480c7b51a914f03bcad7701d75e4dd4
I20240501 07:59:07.838351 9385 ts_tablet_manager.cc:1909] T 8480c7b51a914f03bcad7701d75e4dd4 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Deleting consensus metadata
I20240501 07:59:07.839394 6160 catalog_manager.cc:4830] TS 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131:34471): tablet 8480c7b51a914f03bcad7701d75e4dd4 (table rebalance_test_table_0 [id=6448eafcd04b41cba44dc175507acfc0]) successfully deleted
I20240501 07:59:07.838577 6169 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 0bc706e85ce74cef97a4827ea4d4aa65: rebalancer_dynamic_table_new_name [id=0c9b6b6e953c40078226901a5da69a13] alter complete (version 3)
I20240501 07:59:07.839787 9393 tablet_bootstrap.cc:492] T fb6b46072a8d46e18b53b9f8de702859 P 13d05ebdca8c4f489c597ce550dd773d: No bootstrap required, opened a new log
I20240501 07:59:07.839993 9393 ts_tablet_manager.cc:1367] T fb6b46072a8d46e18b53b9f8de702859 P 13d05ebdca8c4f489c597ce550dd773d: Time spent bootstrapping tablet: real 0.033s user 0.004s sys 0.000s
I20240501 07:59:07.842459 9386 ts_tablet_manager.cc:1888] T e04fe36b8efa4f9584e093279f7addf4 P 8315d22c56f7439a9e1fc82f153f018c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 07:59:07.842603 9386 log.cc:1196] T e04fe36b8efa4f9584e093279f7addf4 P 8315d22c56f7439a9e1fc82f153f018c: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/e04fe36b8efa4f9584e093279f7addf4
I20240501 07:59:07.842947 9386 ts_tablet_manager.cc:1909] T e04fe36b8efa4f9584e093279f7addf4 P 8315d22c56f7439a9e1fc82f153f018c: Deleting consensus metadata
I20240501 07:59:07.843572 9386 tablet_replica.cc:330] T 8480c7b51a914f03bcad7701d75e4dd4 P 8315d22c56f7439a9e1fc82f153f018c: stopping tablet replica
I20240501 07:59:07.843681 6159 catalog_manager.cc:4830] TS 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130:32941): tablet e04fe36b8efa4f9584e093279f7addf4 (table rebalance_test_table_0 [id=6448eafcd04b41cba44dc175507acfc0]) successfully deleted
I20240501 07:59:07.843786 9386 raft_consensus.cc:2229] T 8480c7b51a914f03bcad7701d75e4dd4 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:07.843909 9386 raft_consensus.cc:2258] T 8480c7b51a914f03bcad7701d75e4dd4 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:07.844420 9386 ts_tablet_manager.cc:1875] T 8480c7b51a914f03bcad7701d75e4dd4 P 8315d22c56f7439a9e1fc82f153f018c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:59:07.845290 9392 tablet_bootstrap.cc:492] T fb6b46072a8d46e18b53b9f8de702859 P 74077b61d6f54342bf9736b4a9f2eade: No bootstrap required, opened a new log
I20240501 07:59:07.845782 9392 ts_tablet_manager.cc:1367] T fb6b46072a8d46e18b53b9f8de702859 P 74077b61d6f54342bf9736b4a9f2eade: Time spent bootstrapping tablet: real 0.040s user 0.006s sys 0.000s
I20240501 07:59:07.846552 9386 ts_tablet_manager.cc:1888] T 8480c7b51a914f03bcad7701d75e4dd4 P 8315d22c56f7439a9e1fc82f153f018c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 07:59:07.846673 9386 log.cc:1196] T 8480c7b51a914f03bcad7701d75e4dd4 P 8315d22c56f7439a9e1fc82f153f018c: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/8480c7b51a914f03bcad7701d75e4dd4
I20240501 07:59:07.846968 9386 ts_tablet_manager.cc:1909] T 8480c7b51a914f03bcad7701d75e4dd4 P 8315d22c56f7439a9e1fc82f153f018c: Deleting consensus metadata
I20240501 07:59:07.847659 6159 catalog_manager.cc:4830] TS 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130:32941): tablet 8480c7b51a914f03bcad7701d75e4dd4 (table rebalance_test_table_0 [id=6448eafcd04b41cba44dc175507acfc0]) successfully deleted
I20240501 07:59:07.848395 9388 ts_tablet_manager.cc:1888] T 8480c7b51a914f03bcad7701d75e4dd4 P 8b8b3ad3aa8f4053b2167dd550f2086f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 07:59:07.848924 9388 log.cc:1196] T 8480c7b51a914f03bcad7701d75e4dd4 P 8b8b3ad3aa8f4053b2167dd550f2086f: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/8480c7b51a914f03bcad7701d75e4dd4
I20240501 07:59:07.849234 9388 ts_tablet_manager.cc:1909] T 8480c7b51a914f03bcad7701d75e4dd4 P 8b8b3ad3aa8f4053b2167dd550f2086f: Deleting consensus metadata
I20240501 07:59:07.849898 6157 catalog_manager.cc:4830] TS 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129:46875): tablet 8480c7b51a914f03bcad7701d75e4dd4 (table rebalance_test_table_0 [id=6448eafcd04b41cba44dc175507acfc0]) successfully deleted
I20240501 07:59:07.849855 9392 raft_consensus.cc:348] T fb6b46072a8d46e18b53b9f8de702859 P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } }
I20240501 07:59:07.850128 9392 raft_consensus.cc:374] T fb6b46072a8d46e18b53b9f8de702859 P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:07.850198 9392 raft_consensus.cc:729] T fb6b46072a8d46e18b53b9f8de702859 P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74077b61d6f54342bf9736b4a9f2eade, State: Initialized, Role: FOLLOWER
I20240501 07:59:07.850657 9392 consensus_queue.cc:260] T fb6b46072a8d46e18b53b9f8de702859 P 74077b61d6f54342bf9736b4a9f2eade [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } }
I20240501 07:59:07.851536 9392 ts_tablet_manager.cc:1398] T fb6b46072a8d46e18b53b9f8de702859 P 74077b61d6f54342bf9736b4a9f2eade: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20240501 07:59:07.851835 9393 raft_consensus.cc:348] T fb6b46072a8d46e18b53b9f8de702859 P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } }
I20240501 07:59:07.852042 9393 raft_consensus.cc:374] T fb6b46072a8d46e18b53b9f8de702859 P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:07.852129 9393 raft_consensus.cc:729] T fb6b46072a8d46e18b53b9f8de702859 P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13d05ebdca8c4f489c597ce550dd773d, State: Initialized, Role: FOLLOWER
I20240501 07:59:07.852550 9393 consensus_queue.cc:260] T fb6b46072a8d46e18b53b9f8de702859 P 13d05ebdca8c4f489c597ce550dd773d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } }
I20240501 07:59:07.853351 9393 ts_tablet_manager.cc:1398] T fb6b46072a8d46e18b53b9f8de702859 P 13d05ebdca8c4f489c597ce550dd773d: Time spent starting tablet: real 0.013s user 0.006s sys 0.000s
I20240501 07:59:07.853597 9393 tablet_bootstrap.cc:492] T f88fe033bce14bf0b1b63a52a7ba137e P 13d05ebdca8c4f489c597ce550dd773d: Bootstrap starting.
W20240501 07:59:07.854135 8972 tablet.cc:2369] T fb6b46072a8d46e18b53b9f8de702859 P 13d05ebdca8c4f489c597ce550dd773d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 07:59:07.855331 8971 heartbeater.cc:503] Master 127.0.99.190:38383 was elected leader, sending a full tablet report...
I20240501 07:59:07.855374 9392 tablet_bootstrap.cc:492] T 6292ff0c7e624dd7b24715b83a30fff8 P 74077b61d6f54342bf9736b4a9f2eade: Bootstrap starting.
I20240501 07:59:07.856536 9393 tablet_bootstrap.cc:654] T f88fe033bce14bf0b1b63a52a7ba137e P 13d05ebdca8c4f489c597ce550dd773d: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:07.857904 9392 tablet_bootstrap.cc:654] T 6292ff0c7e624dd7b24715b83a30fff8 P 74077b61d6f54342bf9736b4a9f2eade: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:07.859227 9392 tablet_bootstrap.cc:492] T 6292ff0c7e624dd7b24715b83a30fff8 P 74077b61d6f54342bf9736b4a9f2eade: No bootstrap required, opened a new log
I20240501 07:59:07.859313 9392 ts_tablet_manager.cc:1367] T 6292ff0c7e624dd7b24715b83a30fff8 P 74077b61d6f54342bf9736b4a9f2eade: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 07:59:07.859786 9392 raft_consensus.cc:348] T 6292ff0c7e624dd7b24715b83a30fff8 P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } }
I20240501 07:59:07.859926 9392 raft_consensus.cc:374] T 6292ff0c7e624dd7b24715b83a30fff8 P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:07.859988 9392 raft_consensus.cc:729] T 6292ff0c7e624dd7b24715b83a30fff8 P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74077b61d6f54342bf9736b4a9f2eade, State: Initialized, Role: FOLLOWER
I20240501 07:59:07.860144 9392 consensus_queue.cc:260] T 6292ff0c7e624dd7b24715b83a30fff8 P 74077b61d6f54342bf9736b4a9f2eade [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } }
I20240501 07:59:07.860373 9392 ts_tablet_manager.cc:1398] T 6292ff0c7e624dd7b24715b83a30fff8 P 74077b61d6f54342bf9736b4a9f2eade: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:07.865485 9399 raft_consensus.cc:482] T 6292ff0c7e624dd7b24715b83a30fff8 P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:07.865640 9399 raft_consensus.cc:504] T 6292ff0c7e624dd7b24715b83a30fff8 P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } }
I20240501 07:59:07.867563 9399 leader_election.cc:288] T 6292ff0c7e624dd7b24715b83a30fff8 P 74077b61d6f54342bf9736b4a9f2eade [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149), 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131:34471)
I20240501 07:59:07.869364 9256 tablet_bootstrap.cc:492] T f88fe033bce14bf0b1b63a52a7ba137e P 83f20e84af4c40d09a4804b6b26e697f: No bootstrap required, opened a new log
I20240501 07:59:07.869473 9256 ts_tablet_manager.cc:1367] T f88fe033bce14bf0b1b63a52a7ba137e P 83f20e84af4c40d09a4804b6b26e697f: Time spent bootstrapping tablet: real 0.073s user 0.002s sys 0.000s
I20240501 07:59:07.869904 9256 raft_consensus.cc:348] T f88fe033bce14bf0b1b63a52a7ba137e P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } }
I20240501 07:59:07.870023 9256 raft_consensus.cc:374] T f88fe033bce14bf0b1b63a52a7ba137e P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:07.870049 9256 raft_consensus.cc:729] T f88fe033bce14bf0b1b63a52a7ba137e P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83f20e84af4c40d09a4804b6b26e697f, State: Initialized, Role: FOLLOWER
I20240501 07:59:07.870175 9256 consensus_queue.cc:260] T f88fe033bce14bf0b1b63a52a7ba137e P 83f20e84af4c40d09a4804b6b26e697f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } }
I20240501 07:59:07.870487 9256 ts_tablet_manager.cc:1398] T f88fe033bce14bf0b1b63a52a7ba137e P 83f20e84af4c40d09a4804b6b26e697f: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 07:59:07.870671 9256 tablet_bootstrap.cc:492] T fb6b46072a8d46e18b53b9f8de702859 P 83f20e84af4c40d09a4804b6b26e697f: Bootstrap starting.
I20240501 07:59:07.871418 6577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6292ff0c7e624dd7b24715b83a30fff8" candidate_uuid: "74077b61d6f54342bf9736b4a9f2eade" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" is_pre_election: true
I20240501 07:59:07.871560 9256 tablet_bootstrap.cc:654] T fb6b46072a8d46e18b53b9f8de702859 P 83f20e84af4c40d09a4804b6b26e697f: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:07.871551 6577 raft_consensus.cc:2454] T 6292ff0c7e624dd7b24715b83a30fff8 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74077b61d6f54342bf9736b4a9f2eade in term 0.
I20240501 07:59:07.872344 8730 leader_election.cc:302] T 6292ff0c7e624dd7b24715b83a30fff8 P 74077b61d6f54342bf9736b4a9f2eade [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b164cbbc00f4e8eb3254ddd349a3ff9, 74077b61d6f54342bf9736b4a9f2eade; no voters:
I20240501 07:59:07.873309 9189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6292ff0c7e624dd7b24715b83a30fff8" candidate_uuid: "74077b61d6f54342bf9736b4a9f2eade" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83f20e84af4c40d09a4804b6b26e697f" is_pre_election: true
I20240501 07:59:07.873436 9393 tablet_bootstrap.cc:492] T f88fe033bce14bf0b1b63a52a7ba137e P 13d05ebdca8c4f489c597ce550dd773d: No bootstrap required, opened a new log
I20240501 07:59:07.873442 9189 raft_consensus.cc:2454] T 6292ff0c7e624dd7b24715b83a30fff8 P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74077b61d6f54342bf9736b4a9f2eade in term 0.
I20240501 07:59:07.873543 9393 ts_tablet_manager.cc:1367] T f88fe033bce14bf0b1b63a52a7ba137e P 13d05ebdca8c4f489c597ce550dd773d: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.000s
I20240501 07:59:07.874578 9256 tablet_bootstrap.cc:492] T fb6b46072a8d46e18b53b9f8de702859 P 83f20e84af4c40d09a4804b6b26e697f: No bootstrap required, opened a new log
I20240501 07:59:07.874672 9256 ts_tablet_manager.cc:1367] T fb6b46072a8d46e18b53b9f8de702859 P 83f20e84af4c40d09a4804b6b26e697f: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20240501 07:59:07.874529 9393 raft_consensus.cc:348] T f88fe033bce14bf0b1b63a52a7ba137e P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } }
I20240501 07:59:07.874825 9393 raft_consensus.cc:374] T f88fe033bce14bf0b1b63a52a7ba137e P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:07.875151 9256 raft_consensus.cc:348] T fb6b46072a8d46e18b53b9f8de702859 P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } }
I20240501 07:59:07.875185 9393 raft_consensus.cc:729] T f88fe033bce14bf0b1b63a52a7ba137e P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13d05ebdca8c4f489c597ce550dd773d, State: Initialized, Role: FOLLOWER
I20240501 07:59:07.875260 9256 raft_consensus.cc:374] T fb6b46072a8d46e18b53b9f8de702859 P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:07.875301 9256 raft_consensus.cc:729] T fb6b46072a8d46e18b53b9f8de702859 P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83f20e84af4c40d09a4804b6b26e697f, State: Initialized, Role: FOLLOWER
I20240501 07:59:07.874953 9399 raft_consensus.cc:2781] T 6292ff0c7e624dd7b24715b83a30fff8 P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:07.875406 9399 raft_consensus.cc:482] T 6292ff0c7e624dd7b24715b83a30fff8 P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:07.875423 9256 consensus_queue.cc:260] T fb6b46072a8d46e18b53b9f8de702859 P 83f20e84af4c40d09a4804b6b26e697f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } }
I20240501 07:59:07.875475 9399 raft_consensus.cc:3037] T 6292ff0c7e624dd7b24715b83a30fff8 P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:07.875692 9256 ts_tablet_manager.cc:1398] T fb6b46072a8d46e18b53b9f8de702859 P 83f20e84af4c40d09a4804b6b26e697f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:59:07.877077 9393 consensus_queue.cc:260] T f88fe033bce14bf0b1b63a52a7ba137e P 13d05ebdca8c4f489c597ce550dd773d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } }
I20240501 07:59:07.877856 9393 ts_tablet_manager.cc:1398] T f88fe033bce14bf0b1b63a52a7ba137e P 13d05ebdca8c4f489c597ce550dd773d: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240501 07:59:07.876899 9399 raft_consensus.cc:504] T 6292ff0c7e624dd7b24715b83a30fff8 P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } }
I20240501 07:59:07.878620 9399 leader_election.cc:288] T 6292ff0c7e624dd7b24715b83a30fff8 P 74077b61d6f54342bf9736b4a9f2eade [CANDIDATE]: Term 1 election: Requested vote from peers 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149), 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131:34471)
I20240501 07:59:07.878705 9189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6292ff0c7e624dd7b24715b83a30fff8" candidate_uuid: "74077b61d6f54342bf9736b4a9f2eade" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83f20e84af4c40d09a4804b6b26e697f"
I20240501 07:59:07.878816 9189 raft_consensus.cc:3037] T 6292ff0c7e624dd7b24715b83a30fff8 P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:07.879868 9189 raft_consensus.cc:2454] T 6292ff0c7e624dd7b24715b83a30fff8 P 83f20e84af4c40d09a4804b6b26e697f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74077b61d6f54342bf9736b4a9f2eade in term 1.
I20240501 07:59:07.880187 8728 leader_election.cc:302] T 6292ff0c7e624dd7b24715b83a30fff8 P 74077b61d6f54342bf9736b4a9f2eade [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74077b61d6f54342bf9736b4a9f2eade, 83f20e84af4c40d09a4804b6b26e697f; no voters:
I20240501 07:59:07.880548 9399 raft_consensus.cc:2781] T 6292ff0c7e624dd7b24715b83a30fff8 P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:07.880647 6577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6292ff0c7e624dd7b24715b83a30fff8" candidate_uuid: "74077b61d6f54342bf9736b4a9f2eade" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9"
I20240501 07:59:07.880785 6577 raft_consensus.cc:3037] T 6292ff0c7e624dd7b24715b83a30fff8 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:07.880815 9399 raft_consensus.cc:686] T 6292ff0c7e624dd7b24715b83a30fff8 P 74077b61d6f54342bf9736b4a9f2eade [term 1 LEADER]: Becoming Leader. State: Replica: 74077b61d6f54342bf9736b4a9f2eade, State: Running, Role: LEADER
I20240501 07:59:07.881192 9399 consensus_queue.cc:237] T 6292ff0c7e624dd7b24715b83a30fff8 P 74077b61d6f54342bf9736b4a9f2eade [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } }
I20240501 07:59:07.881793 6577 raft_consensus.cc:2454] T 6292ff0c7e624dd7b24715b83a30fff8 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74077b61d6f54342bf9736b4a9f2eade in term 1.
I20240501 07:59:07.884243 6169 catalog_manager.cc:5484] T 6292ff0c7e624dd7b24715b83a30fff8 P 74077b61d6f54342bf9736b4a9f2eade reported cstate change: term changed from 0 to 1, leader changed from <none> to 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133). New cstate: current_term: 1 leader_uuid: "74077b61d6f54342bf9736b4a9f2eade" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } health_report { overall_health: HEALTHY } } }
W20240501 07:59:07.908315 8841 tablet.cc:2369] T 6292ff0c7e624dd7b24715b83a30fff8 P 74077b61d6f54342bf9736b4a9f2eade: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 07:59:07.918762 9400 raft_consensus.cc:482] T fb6b46072a8d46e18b53b9f8de702859 P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:07.918928 9400 raft_consensus.cc:504] T fb6b46072a8d46e18b53b9f8de702859 P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } }
I20240501 07:59:07.919945 9400 leader_election.cc:288] T fb6b46072a8d46e18b53b9f8de702859 P 13d05ebdca8c4f489c597ce550dd773d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149), 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133:39859)
I20240501 07:59:07.924680 8794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb6b46072a8d46e18b53b9f8de702859" candidate_uuid: "13d05ebdca8c4f489c597ce550dd773d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74077b61d6f54342bf9736b4a9f2eade" is_pre_election: true
I20240501 07:59:07.924665 9189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb6b46072a8d46e18b53b9f8de702859" candidate_uuid: "13d05ebdca8c4f489c597ce550dd773d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83f20e84af4c40d09a4804b6b26e697f" is_pre_election: true
I20240501 07:59:07.924860 9189 raft_consensus.cc:2454] T fb6b46072a8d46e18b53b9f8de702859 P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 13d05ebdca8c4f489c597ce550dd773d in term 0.
I20240501 07:59:07.924901 8794 raft_consensus.cc:2454] T fb6b46072a8d46e18b53b9f8de702859 P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 13d05ebdca8c4f489c597ce550dd773d in term 0.
I20240501 07:59:07.925314 8859 leader_election.cc:302] T fb6b46072a8d46e18b53b9f8de702859 P 13d05ebdca8c4f489c597ce550dd773d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 13d05ebdca8c4f489c597ce550dd773d, 74077b61d6f54342bf9736b4a9f2eade, 83f20e84af4c40d09a4804b6b26e697f; no voters:
I20240501 07:59:07.925609 9400 raft_consensus.cc:2781] T fb6b46072a8d46e18b53b9f8de702859 P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:07.925710 9400 raft_consensus.cc:482] T fb6b46072a8d46e18b53b9f8de702859 P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:07.925765 9400 raft_consensus.cc:3037] T fb6b46072a8d46e18b53b9f8de702859 P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:07.926978 9400 raft_consensus.cc:504] T fb6b46072a8d46e18b53b9f8de702859 P 13d05ebdca8c4f489c597ce550dd773d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } }
I20240501 07:59:07.927384 9400 leader_election.cc:288] T fb6b46072a8d46e18b53b9f8de702859 P 13d05ebdca8c4f489c597ce550dd773d [CANDIDATE]: Term 1 election: Requested vote from peers 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149), 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133:39859)
I20240501 07:59:07.927673 9189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb6b46072a8d46e18b53b9f8de702859" candidate_uuid: "13d05ebdca8c4f489c597ce550dd773d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83f20e84af4c40d09a4804b6b26e697f"
I20240501 07:59:07.927716 8794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb6b46072a8d46e18b53b9f8de702859" candidate_uuid: "13d05ebdca8c4f489c597ce550dd773d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74077b61d6f54342bf9736b4a9f2eade"
I20240501 07:59:07.927806 9189 raft_consensus.cc:3037] T fb6b46072a8d46e18b53b9f8de702859 P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:07.927824 8794 raft_consensus.cc:3037] T fb6b46072a8d46e18b53b9f8de702859 P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:07.928783 9189 raft_consensus.cc:2454] T fb6b46072a8d46e18b53b9f8de702859 P 83f20e84af4c40d09a4804b6b26e697f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13d05ebdca8c4f489c597ce550dd773d in term 1.
I20240501 07:59:07.928783 8794 raft_consensus.cc:2454] T fb6b46072a8d46e18b53b9f8de702859 P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13d05ebdca8c4f489c597ce550dd773d in term 1.
I20240501 07:59:07.929273 8858 leader_election.cc:302] T fb6b46072a8d46e18b53b9f8de702859 P 13d05ebdca8c4f489c597ce550dd773d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13d05ebdca8c4f489c597ce550dd773d, 74077b61d6f54342bf9736b4a9f2eade; no voters:
I20240501 07:59:07.929653 9400 raft_consensus.cc:2781] T fb6b46072a8d46e18b53b9f8de702859 P 13d05ebdca8c4f489c597ce550dd773d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:07.929890 9400 raft_consensus.cc:686] T fb6b46072a8d46e18b53b9f8de702859 P 13d05ebdca8c4f489c597ce550dd773d [term 1 LEADER]: Becoming Leader. State: Replica: 13d05ebdca8c4f489c597ce550dd773d, State: Running, Role: LEADER
I20240501 07:59:07.930223 9400 consensus_queue.cc:237] T fb6b46072a8d46e18b53b9f8de702859 P 13d05ebdca8c4f489c597ce550dd773d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } }
I20240501 07:59:07.933248 6169 catalog_manager.cc:5484] T fb6b46072a8d46e18b53b9f8de702859 P 13d05ebdca8c4f489c597ce550dd773d reported cstate change: term changed from 0 to 1, leader changed from <none> to 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134). New cstate: current_term: 1 leader_uuid: "13d05ebdca8c4f489c597ce550dd773d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } health_report { overall_health: HEALTHY } } }
I20240501 07:59:07.939837 6169 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55950:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 07:59:07.939981 6169 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55950:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 07:59:07.942098 6169 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 0bc706e85ce74cef97a4827ea4d4aa65: Sending DeleteTablet for 3 replicas of tablet 733ab980b9a442ed90b5a10657b8285e
I20240501 07:59:07.942698 9036 tablet_service.cc:1514] Processing DeleteTablet for tablet 733ab980b9a442ed90b5a10657b8285e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:07 UTC) from {username='slave'} at 127.0.0.1:44236
I20240501 07:59:07.943138 9433 tablet_replica.cc:330] T 733ab980b9a442ed90b5a10657b8285e P a505e7cb133c4392bfda8a4a20874ae9: stopping tablet replica
I20240501 07:59:07.943279 6294 tablet_service.cc:1514] Processing DeleteTablet for tablet 733ab980b9a442ed90b5a10657b8285e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:07 UTC) from {username='slave'} at 127.0.0.1:45480
I20240501 07:59:07.943324 9169 tablet_service.cc:1514] Processing DeleteTablet for tablet 733ab980b9a442ed90b5a10657b8285e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:07 UTC) from {username='slave'} at 127.0.0.1:33594
I20240501 07:59:07.943467 9433 raft_consensus.cc:2229] T 733ab980b9a442ed90b5a10657b8285e P a505e7cb133c4392bfda8a4a20874ae9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:07.943670 9433 raft_consensus.cc:2258] T 733ab980b9a442ed90b5a10657b8285e P a505e7cb133c4392bfda8a4a20874ae9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:07.943739 9388 tablet_replica.cc:330] T 733ab980b9a442ed90b5a10657b8285e P 8b8b3ad3aa8f4053b2167dd550f2086f: stopping tablet replica
I20240501 07:59:07.943796 9434 tablet_replica.cc:330] T 733ab980b9a442ed90b5a10657b8285e P 83f20e84af4c40d09a4804b6b26e697f: stopping tablet replica
I20240501 07:59:07.943884 9388 raft_consensus.cc:2229] T 733ab980b9a442ed90b5a10657b8285e P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 LEADER]: Raft consensus shutting down.
I20240501 07:59:07.944043 9434 raft_consensus.cc:2229] T 733ab980b9a442ed90b5a10657b8285e P 83f20e84af4c40d09a4804b6b26e697f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:07.944083 9388 raft_consensus.cc:2258] T 733ab980b9a442ed90b5a10657b8285e P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:07.944229 9434 raft_consensus.cc:2258] T 733ab980b9a442ed90b5a10657b8285e P 83f20e84af4c40d09a4804b6b26e697f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:07.944669 9388 ts_tablet_manager.cc:1875] T 733ab980b9a442ed90b5a10657b8285e P 8b8b3ad3aa8f4053b2167dd550f2086f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:59:07.945292 9434 ts_tablet_manager.cc:1875] T 733ab980b9a442ed90b5a10657b8285e P 83f20e84af4c40d09a4804b6b26e697f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:59:07.946851 9388 ts_tablet_manager.cc:1888] T 733ab980b9a442ed90b5a10657b8285e P 8b8b3ad3aa8f4053b2167dd550f2086f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:59:07.946923 9388 log.cc:1196] T 733ab980b9a442ed90b5a10657b8285e P 8b8b3ad3aa8f4053b2167dd550f2086f: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/733ab980b9a442ed90b5a10657b8285e
I20240501 07:59:07.947188 9388 ts_tablet_manager.cc:1909] T 733ab980b9a442ed90b5a10657b8285e P 8b8b3ad3aa8f4053b2167dd550f2086f: Deleting consensus metadata
I20240501 07:59:07.947845 9434 ts_tablet_manager.cc:1888] T 733ab980b9a442ed90b5a10657b8285e P 83f20e84af4c40d09a4804b6b26e697f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:59:07.947921 9434 log.cc:1196] T 733ab980b9a442ed90b5a10657b8285e P 83f20e84af4c40d09a4804b6b26e697f: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/733ab980b9a442ed90b5a10657b8285e
I20240501 07:59:07.948220 9434 ts_tablet_manager.cc:1909] T 733ab980b9a442ed90b5a10657b8285e P 83f20e84af4c40d09a4804b6b26e697f: Deleting consensus metadata
I20240501 07:59:07.948400 6157 catalog_manager.cc:4830] TS 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129:46875): tablet 733ab980b9a442ed90b5a10657b8285e (table rebalancer_dynamic_table_new_name [id=0c9b6b6e953c40078226901a5da69a13]) successfully deleted
I20240501 07:59:07.949295 9433 ts_tablet_manager.cc:1875] T 733ab980b9a442ed90b5a10657b8285e P a505e7cb133c4392bfda8a4a20874ae9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:59:07.949302 6158 catalog_manager.cc:4830] TS 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): tablet 733ab980b9a442ed90b5a10657b8285e (table rebalancer_dynamic_table_new_name [id=0c9b6b6e953c40078226901a5da69a13]) successfully deleted
I20240501 07:59:07.954201 9433 ts_tablet_manager.cc:1888] T 733ab980b9a442ed90b5a10657b8285e P a505e7cb133c4392bfda8a4a20874ae9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:59:07.954334 9255 raft_consensus.cc:482] T f88fe033bce14bf0b1b63a52a7ba137e P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:07.954435 9433 log.cc:1196] T 733ab980b9a442ed90b5a10657b8285e P a505e7cb133c4392bfda8a4a20874ae9: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/733ab980b9a442ed90b5a10657b8285e
I20240501 07:59:07.954433 9255 raft_consensus.cc:504] T f88fe033bce14bf0b1b63a52a7ba137e P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } }
I20240501 07:59:07.954854 9433 ts_tablet_manager.cc:1909] T 733ab980b9a442ed90b5a10657b8285e P a505e7cb133c4392bfda8a4a20874ae9: Deleting consensus metadata
I20240501 07:59:07.955830 9255 leader_election.cc:288] T f88fe033bce14bf0b1b63a52a7ba137e P a505e7cb133c4392bfda8a4a20874ae9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134:40141), 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149)
I20240501 07:59:07.955999 6158 catalog_manager.cc:4830] TS a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): tablet 733ab980b9a442ed90b5a10657b8285e (table rebalancer_dynamic_table_new_name [id=0c9b6b6e953c40078226901a5da69a13]) successfully deleted
I20240501 07:59:07.960284 9189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f88fe033bce14bf0b1b63a52a7ba137e" candidate_uuid: "a505e7cb133c4392bfda8a4a20874ae9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83f20e84af4c40d09a4804b6b26e697f" is_pre_election: true
I20240501 07:59:07.960444 9189 raft_consensus.cc:2454] T f88fe033bce14bf0b1b63a52a7ba137e P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a505e7cb133c4392bfda8a4a20874ae9 in term 0.
I20240501 07:59:07.961241 8990 leader_election.cc:302] T f88fe033bce14bf0b1b63a52a7ba137e P a505e7cb133c4392bfda8a4a20874ae9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83f20e84af4c40d09a4804b6b26e697f, a505e7cb133c4392bfda8a4a20874ae9; no voters:
I20240501 07:59:07.962230 9255 raft_consensus.cc:2781] T f88fe033bce14bf0b1b63a52a7ba137e P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:07.962286 9255 raft_consensus.cc:482] T f88fe033bce14bf0b1b63a52a7ba137e P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:07.962327 9255 raft_consensus.cc:3037] T f88fe033bce14bf0b1b63a52a7ba137e P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:07.962540 8925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f88fe033bce14bf0b1b63a52a7ba137e" candidate_uuid: "a505e7cb133c4392bfda8a4a20874ae9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13d05ebdca8c4f489c597ce550dd773d" is_pre_election: true
I20240501 07:59:07.962780 8925 raft_consensus.cc:2454] T f88fe033bce14bf0b1b63a52a7ba137e P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a505e7cb133c4392bfda8a4a20874ae9 in term 0.
I20240501 07:59:07.963330 9255 raft_consensus.cc:504] T f88fe033bce14bf0b1b63a52a7ba137e P a505e7cb133c4392bfda8a4a20874ae9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } }
I20240501 07:59:07.963634 9255 leader_election.cc:288] T f88fe033bce14bf0b1b63a52a7ba137e P a505e7cb133c4392bfda8a4a20874ae9 [CANDIDATE]: Term 1 election: Requested vote from peers 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134:40141), 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149)
I20240501 07:59:07.963960 9189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f88fe033bce14bf0b1b63a52a7ba137e" candidate_uuid: "a505e7cb133c4392bfda8a4a20874ae9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83f20e84af4c40d09a4804b6b26e697f"
I20240501 07:59:07.963965 8925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f88fe033bce14bf0b1b63a52a7ba137e" candidate_uuid: "a505e7cb133c4392bfda8a4a20874ae9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13d05ebdca8c4f489c597ce550dd773d"
I20240501 07:59:07.964066 9189 raft_consensus.cc:3037] T f88fe033bce14bf0b1b63a52a7ba137e P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:07.964087 8925 raft_consensus.cc:3037] T f88fe033bce14bf0b1b63a52a7ba137e P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:07.964891 9189 raft_consensus.cc:2454] T f88fe033bce14bf0b1b63a52a7ba137e P 83f20e84af4c40d09a4804b6b26e697f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a505e7cb133c4392bfda8a4a20874ae9 in term 1.
I20240501 07:59:07.965173 8990 leader_election.cc:302] T f88fe033bce14bf0b1b63a52a7ba137e P a505e7cb133c4392bfda8a4a20874ae9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83f20e84af4c40d09a4804b6b26e697f, a505e7cb133c4392bfda8a4a20874ae9; no voters:
I20240501 07:59:07.965340 9255 raft_consensus.cc:2781] T f88fe033bce14bf0b1b63a52a7ba137e P a505e7cb133c4392bfda8a4a20874ae9 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:07.965518 8925 raft_consensus.cc:2454] T f88fe033bce14bf0b1b63a52a7ba137e P 13d05ebdca8c4f489c597ce550dd773d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a505e7cb133c4392bfda8a4a20874ae9 in term 1.
I20240501 07:59:07.965566 9255 raft_consensus.cc:686] T f88fe033bce14bf0b1b63a52a7ba137e P a505e7cb133c4392bfda8a4a20874ae9 [term 1 LEADER]: Becoming Leader. State: Replica: a505e7cb133c4392bfda8a4a20874ae9, State: Running, Role: LEADER
I20240501 07:59:07.965838 9255 consensus_queue.cc:237] T f88fe033bce14bf0b1b63a52a7ba137e P a505e7cb133c4392bfda8a4a20874ae9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } }
I20240501 07:59:07.968302 6166 catalog_manager.cc:5484] T f88fe033bce14bf0b1b63a52a7ba137e P a505e7cb133c4392bfda8a4a20874ae9 reported cstate change: term changed from 0 to 1, leader changed from <none> to a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135). New cstate: current_term: 1 leader_uuid: "a505e7cb133c4392bfda8a4a20874ae9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } health_report { overall_health: HEALTHY } } }
I20240501 07:59:08.014603 8774 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:59:08.015337 6687 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 07:59:08.022491 9036 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:59:08.030966 6556 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 07:59:08.034947 9169 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 07:59:08.035885 8905 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:59:08.043619 6166 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55950:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 07:59:08.049834 6426 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 07:59:08.052678 8774 tablet_service.cc:1467] Processing CreateTablet for tablet 4cdabbbfd1df4a5a9683e3c8e569d624 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=5a2c4b25a3944383893b9fd726c21cad]), partition=
I20240501 07:59:08.052901 9169 tablet_service.cc:1467] Processing CreateTablet for tablet 4cdabbbfd1df4a5a9683e3c8e569d624 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=5a2c4b25a3944383893b9fd726c21cad]), partition=
I20240501 07:59:08.053156 9169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4cdabbbfd1df4a5a9683e3c8e569d624. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:08.053159 8774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4cdabbbfd1df4a5a9683e3c8e569d624. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:08.055392 9392 tablet_bootstrap.cc:492] T 4cdabbbfd1df4a5a9683e3c8e569d624 P 74077b61d6f54342bf9736b4a9f2eade: Bootstrap starting.
I20240501 07:59:08.055392 9256 tablet_bootstrap.cc:492] T 4cdabbbfd1df4a5a9683e3c8e569d624 P 83f20e84af4c40d09a4804b6b26e697f: Bootstrap starting.
I20240501 07:59:08.056201 9256 tablet_bootstrap.cc:654] T 4cdabbbfd1df4a5a9683e3c8e569d624 P 83f20e84af4c40d09a4804b6b26e697f: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:08.056201 9392 tablet_bootstrap.cc:654] T 4cdabbbfd1df4a5a9683e3c8e569d624 P 74077b61d6f54342bf9736b4a9f2eade: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:08.058187 8905 tablet_service.cc:1467] Processing CreateTablet for tablet 4cdabbbfd1df4a5a9683e3c8e569d624 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=5a2c4b25a3944383893b9fd726c21cad]), partition=
I20240501 07:59:08.058427 8905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4cdabbbfd1df4a5a9683e3c8e569d624. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:08.060557 9393 tablet_bootstrap.cc:492] T 4cdabbbfd1df4a5a9683e3c8e569d624 P 13d05ebdca8c4f489c597ce550dd773d: Bootstrap starting.
I20240501 07:59:08.061499 9393 tablet_bootstrap.cc:654] T 4cdabbbfd1df4a5a9683e3c8e569d624 P 13d05ebdca8c4f489c597ce550dd773d: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:08.065554 6294 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 07:59:08.068135 9256 tablet_bootstrap.cc:492] T 4cdabbbfd1df4a5a9683e3c8e569d624 P 83f20e84af4c40d09a4804b6b26e697f: No bootstrap required, opened a new log
I20240501 07:59:08.068233 9256 ts_tablet_manager.cc:1367] T 4cdabbbfd1df4a5a9683e3c8e569d624 P 83f20e84af4c40d09a4804b6b26e697f: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.000s
I20240501 07:59:08.068650 9256 raft_consensus.cc:348] T 4cdabbbfd1df4a5a9683e3c8e569d624 P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } }
I20240501 07:59:08.068776 9256 raft_consensus.cc:374] T 4cdabbbfd1df4a5a9683e3c8e569d624 P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:08.068804 9256 raft_consensus.cc:729] T 4cdabbbfd1df4a5a9683e3c8e569d624 P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83f20e84af4c40d09a4804b6b26e697f, State: Initialized, Role: FOLLOWER
I20240501 07:59:08.068912 9256 consensus_queue.cc:260] T 4cdabbbfd1df4a5a9683e3c8e569d624 P 83f20e84af4c40d09a4804b6b26e697f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } }
I20240501 07:59:08.069167 9256 ts_tablet_manager.cc:1398] T 4cdabbbfd1df4a5a9683e3c8e569d624 P 83f20e84af4c40d09a4804b6b26e697f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:08.073742 9393 tablet_bootstrap.cc:492] T 4cdabbbfd1df4a5a9683e3c8e569d624 P 13d05ebdca8c4f489c597ce550dd773d: No bootstrap required, opened a new log
I20240501 07:59:08.073850 9393 ts_tablet_manager.cc:1367] T 4cdabbbfd1df4a5a9683e3c8e569d624 P 13d05ebdca8c4f489c597ce550dd773d: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20240501 07:59:08.074295 9393 raft_consensus.cc:348] T 4cdabbbfd1df4a5a9683e3c8e569d624 P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } }
I20240501 07:59:08.074405 9393 raft_consensus.cc:374] T 4cdabbbfd1df4a5a9683e3c8e569d624 P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:08.074446 9393 raft_consensus.cc:729] T 4cdabbbfd1df4a5a9683e3c8e569d624 P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13d05ebdca8c4f489c597ce550dd773d, State: Initialized, Role: FOLLOWER
I20240501 07:59:08.074599 9393 consensus_queue.cc:260] T 4cdabbbfd1df4a5a9683e3c8e569d624 P 13d05ebdca8c4f489c597ce550dd773d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } }
I20240501 07:59:08.074891 9393 ts_tablet_manager.cc:1398] T 4cdabbbfd1df4a5a9683e3c8e569d624 P 13d05ebdca8c4f489c597ce550dd773d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:08.079180 9392 tablet_bootstrap.cc:492] T 4cdabbbfd1df4a5a9683e3c8e569d624 P 74077b61d6f54342bf9736b4a9f2eade: No bootstrap required, opened a new log
I20240501 07:59:08.079282 9392 ts_tablet_manager.cc:1367] T 4cdabbbfd1df4a5a9683e3c8e569d624 P 74077b61d6f54342bf9736b4a9f2eade: Time spent bootstrapping tablet: real 0.024s user 0.002s sys 0.000s
I20240501 07:59:08.079777 9392 raft_consensus.cc:348] T 4cdabbbfd1df4a5a9683e3c8e569d624 P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } }
I20240501 07:59:08.079892 9392 raft_consensus.cc:374] T 4cdabbbfd1df4a5a9683e3c8e569d624 P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:08.079921 9392 raft_consensus.cc:729] T 4cdabbbfd1df4a5a9683e3c8e569d624 P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74077b61d6f54342bf9736b4a9f2eade, State: Initialized, Role: FOLLOWER
I20240501 07:59:08.080044 9392 consensus_queue.cc:260] T 4cdabbbfd1df4a5a9683e3c8e569d624 P 74077b61d6f54342bf9736b4a9f2eade [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } }
I20240501 07:59:08.080324 9392 ts_tablet_manager.cc:1398] T 4cdabbbfd1df4a5a9683e3c8e569d624 P 74077b61d6f54342bf9736b4a9f2eade: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:59:08.139101 6708 consensus_queue.cc:237] T 26713549654249b09f820a31ee19e2b7 P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { replace: true } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: true } }
I20240501 07:59:08.140689 6313 raft_consensus.cc:1261] T 26713549654249b09f820a31ee19e2b7 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Refusing update from remote peer 1f9cb72be98941b5a8e504cbe34132b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:59:08.140677 6444 raft_consensus.cc:1261] T 26713549654249b09f820a31ee19e2b7 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Refusing update from remote peer 1f9cb72be98941b5a8e504cbe34132b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:59:08.141242 9479 consensus_queue.cc:1035] T 26713549654249b09f820a31ee19e2b7 P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:08.141445 8710 consensus_queue.cc:1035] T 26713549654249b09f820a31ee19e2b7 P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:08.143401 8710 raft_consensus.cc:2932] T 26713549654249b09f820a31ee19e2b7 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { replace: true } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: true } } }
I20240501 07:59:08.144274 6444 raft_consensus.cc:2932] T 26713549654249b09f820a31ee19e2b7 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { replace: true } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: true } } }
W20240501 07:59:08.145665 6642 consensus_peers.cc:485] T 26713549654249b09f820a31ee19e2b7 P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 26713549654249b09f820a31ee19e2b7. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:08.147186 6313 raft_consensus.cc:2932] T 26713549654249b09f820a31ee19e2b7 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { replace: true } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: true } } }
I20240501 07:59:08.148288 6166 catalog_manager.cc:5484] T 26713549654249b09f820a31ee19e2b7 P 1f9cb72be98941b5a8e504cbe34132b9 reported cstate change: config changed from index -1 to 4, NON_VOTER a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "1f9cb72be98941b5a8e504cbe34132b9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 07:59:08.156193 6444 consensus_queue.cc:237] T 271e3c5cf9934eaead7fa1ba942084be P 8315d22c56f7439a9e1fc82f153f018c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: true } }
I20240501 07:59:08.157603 6708 raft_consensus.cc:1261] T 271e3c5cf9934eaead7fa1ba942084be P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Refusing update from remote peer 8315d22c56f7439a9e1fc82f153f018c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:59:08.157645 6577 raft_consensus.cc:1261] T 271e3c5cf9934eaead7fa1ba942084be P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Refusing update from remote peer 8315d22c56f7439a9e1fc82f153f018c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:59:08.157941 8512 consensus_queue.cc:1035] T 271e3c5cf9934eaead7fa1ba942084be P 8315d22c56f7439a9e1fc82f153f018c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:08.158295 8512 consensus_queue.cc:1035] T 271e3c5cf9934eaead7fa1ba942084be P 8315d22c56f7439a9e1fc82f153f018c [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:08.160265 9116 raft_consensus.cc:2932] T 271e3c5cf9934eaead7fa1ba942084be P 8315d22c56f7439a9e1fc82f153f018c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: true } } }
I20240501 07:59:08.160717 6708 raft_consensus.cc:2932] T 271e3c5cf9934eaead7fa1ba942084be P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: true } } }
I20240501 07:59:08.160755 6577 raft_consensus.cc:2932] T 271e3c5cf9934eaead7fa1ba942084be P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: true } } }
W20240501 07:59:08.163662 6379 consensus_peers.cc:485] T 271e3c5cf9934eaead7fa1ba942084be P 8315d22c56f7439a9e1fc82f153f018c -> Peer 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133:39859): Couldn't send request to peer 74077b61d6f54342bf9736b4a9f2eade. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 271e3c5cf9934eaead7fa1ba942084be. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:08.164091 6168 catalog_manager.cc:5484] T 271e3c5cf9934eaead7fa1ba942084be P 8315d22c56f7439a9e1fc82f153f018c reported cstate change: config changed from index -1 to 4, NON_VOTER 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "8315d22c56f7439a9e1fc82f153f018c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 07:59:08.172657 6444 consensus_queue.cc:237] T 7d63a75f2194484f8a59709a209f00ea P 8315d22c56f7439a9e1fc82f153f018c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { replace: true } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: true } }
I20240501 07:59:08.174265 6577 raft_consensus.cc:1261] T 7d63a75f2194484f8a59709a209f00ea P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Refusing update from remote peer 8315d22c56f7439a9e1fc82f153f018c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:59:08.174355 6708 raft_consensus.cc:1261] T 7d63a75f2194484f8a59709a209f00ea P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Refusing update from remote peer 8315d22c56f7439a9e1fc82f153f018c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:59:08.174741 8512 consensus_queue.cc:1035] T 7d63a75f2194484f8a59709a209f00ea P 8315d22c56f7439a9e1fc82f153f018c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:08.174969 9489 consensus_queue.cc:1035] T 7d63a75f2194484f8a59709a209f00ea P 8315d22c56f7439a9e1fc82f153f018c [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:08.176661 8512 raft_consensus.cc:2932] T 7d63a75f2194484f8a59709a209f00ea P 8315d22c56f7439a9e1fc82f153f018c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { replace: true } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: true } } }
I20240501 07:59:08.178799 6577 raft_consensus.cc:2932] T 7d63a75f2194484f8a59709a209f00ea P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { replace: true } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: true } } }
W20240501 07:59:08.179303 6381 consensus_peers.cc:485] T 7d63a75f2194484f8a59709a209f00ea P 8315d22c56f7439a9e1fc82f153f018c -> Peer 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134:40141): Couldn't send request to peer 13d05ebdca8c4f489c597ce550dd773d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7d63a75f2194484f8a59709a209f00ea. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:08.180050 6172 catalog_manager.cc:5484] T 7d63a75f2194484f8a59709a209f00ea P 8315d22c56f7439a9e1fc82f153f018c reported cstate change: config changed from index -1 to 4, NON_VOTER 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "8315d22c56f7439a9e1fc82f153f018c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 07:59:08.180078 6708 raft_consensus.cc:2932] T 7d63a75f2194484f8a59709a209f00ea P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { replace: true } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: true } } }
I20240501 07:59:08.189992 6575 consensus_queue.cc:237] T d3f2250a62c6429e886753aaffcbe883 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: true } }
I20240501 07:59:08.191583 6708 raft_consensus.cc:1261] T d3f2250a62c6429e886753aaffcbe883 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Refusing update from remote peer 3b164cbbc00f4e8eb3254ddd349a3ff9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:59:08.191969 6444 raft_consensus.cc:1261] T d3f2250a62c6429e886753aaffcbe883 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Refusing update from remote peer 3b164cbbc00f4e8eb3254ddd349a3ff9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:59:08.192016 9507 consensus_queue.cc:1035] T d3f2250a62c6429e886753aaffcbe883 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:08.193687 9507 consensus_queue.cc:1035] T d3f2250a62c6429e886753aaffcbe883 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:08.195300 8680 raft_consensus.cc:2932] T d3f2250a62c6429e886753aaffcbe883 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: true } } }
W20240501 07:59:08.195555 6511 consensus_peers.cc:485] T d3f2250a62c6429e886753aaffcbe883 P 3b164cbbc00f4e8eb3254ddd349a3ff9 -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d3f2250a62c6429e886753aaffcbe883. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:08.197197 6708 raft_consensus.cc:2932] T d3f2250a62c6429e886753aaffcbe883 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: true } } }
I20240501 07:59:08.198249 6168 catalog_manager.cc:5484] T d3f2250a62c6429e886753aaffcbe883 P 3b164cbbc00f4e8eb3254ddd349a3ff9 reported cstate change: config changed from index -1 to 4, NON_VOTER 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 07:59:08.199669 6444 raft_consensus.cc:2932] T d3f2250a62c6429e886753aaffcbe883 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: true } } }
I20240501 07:59:08.215286 6708 consensus_queue.cc:237] T 810a2791eb04415dae47e85db87352cb P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: true } }
I20240501 07:59:08.216933 6575 raft_consensus.cc:1261] T 810a2791eb04415dae47e85db87352cb P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Refusing update from remote peer 1f9cb72be98941b5a8e504cbe34132b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:59:08.217415 9479 consensus_queue.cc:1035] T 810a2791eb04415dae47e85db87352cb P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 07:59:08.217751 6642 consensus_peers.cc:485] T 810a2791eb04415dae47e85db87352cb P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 810a2791eb04415dae47e85db87352cb. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:08.218523 6313 raft_consensus.cc:1261] T 810a2791eb04415dae47e85db87352cb P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Refusing update from remote peer 1f9cb72be98941b5a8e504cbe34132b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:59:08.218955 9398 consensus_queue.cc:1035] T 810a2791eb04415dae47e85db87352cb P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:08.220727 9479 raft_consensus.cc:2932] T 810a2791eb04415dae47e85db87352cb P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: true } } }
I20240501 07:59:08.221199 6313 raft_consensus.cc:2932] T 810a2791eb04415dae47e85db87352cb P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: true } } }
I20240501 07:59:08.224465 6166 catalog_manager.cc:5484] T 810a2791eb04415dae47e85db87352cb P 1f9cb72be98941b5a8e504cbe34132b9 reported cstate change: config changed from index -1 to 4, NON_VOTER a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "1f9cb72be98941b5a8e504cbe34132b9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 07:59:08.227530 6575 raft_consensus.cc:2932] T 810a2791eb04415dae47e85db87352cb P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: true } } }
I20240501 07:59:08.234826 6444 consensus_queue.cc:237] T 6dc929e2a6ae45bd8b1f297583636bb6 P 8315d22c56f7439a9e1fc82f153f018c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: true } }
I20240501 07:59:08.236301 6575 raft_consensus.cc:1261] T 6dc929e2a6ae45bd8b1f297583636bb6 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Refusing update from remote peer 8315d22c56f7439a9e1fc82f153f018c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:59:08.236595 6313 raft_consensus.cc:1261] T 6dc929e2a6ae45bd8b1f297583636bb6 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Refusing update from remote peer 8315d22c56f7439a9e1fc82f153f018c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:59:08.236881 9489 consensus_queue.cc:1035] T 6dc929e2a6ae45bd8b1f297583636bb6 P 8315d22c56f7439a9e1fc82f153f018c [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 07:59:08.237110 6379 consensus_peers.cc:485] T 6dc929e2a6ae45bd8b1f297583636bb6 P 8315d22c56f7439a9e1fc82f153f018c -> Peer 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133:39859): Couldn't send request to peer 74077b61d6f54342bf9736b4a9f2eade. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6dc929e2a6ae45bd8b1f297583636bb6. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:08.237275 9489 consensus_queue.cc:1035] T 6dc929e2a6ae45bd8b1f297583636bb6 P 8315d22c56f7439a9e1fc82f153f018c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:08.238878 9489 raft_consensus.cc:2932] T 6dc929e2a6ae45bd8b1f297583636bb6 P 8315d22c56f7439a9e1fc82f153f018c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: true } } }
I20240501 07:59:08.240588 6313 raft_consensus.cc:2932] T 6dc929e2a6ae45bd8b1f297583636bb6 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: true } } }
I20240501 07:59:08.241231 6575 raft_consensus.cc:2932] T 6dc929e2a6ae45bd8b1f297583636bb6 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: true } } }
I20240501 07:59:08.244208 6166 catalog_manager.cc:5484] T 6dc929e2a6ae45bd8b1f297583636bb6 P 3b164cbbc00f4e8eb3254ddd349a3ff9 reported cstate change: config changed from index -1 to 4, NON_VOTER 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "8315d22c56f7439a9e1fc82f153f018c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: true } } }
I20240501 07:59:08.252153 6168 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55950:
table { table_name: "rebalance_test_table_2" } modify_external_catalogs: true
I20240501 07:59:08.252318 6168 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55950:
table { table_name: "rebalance_test_table_2" } modify_external_catalogs: true
I20240501 07:59:08.252624 6313 consensus_queue.cc:237] T ec2d3ccf0937430f947f3e2bde0506c5 P 8b8b3ad3aa8f4053b2167dd550f2086f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { replace: true } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: true } }
I20240501 07:59:08.254354 6575 raft_consensus.cc:1261] T ec2d3ccf0937430f947f3e2bde0506c5 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Refusing update from remote peer 8b8b3ad3aa8f4053b2167dd550f2086f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:59:08.254758 8447 consensus_queue.cc:1035] T ec2d3ccf0937430f947f3e2bde0506c5 P 8b8b3ad3aa8f4053b2167dd550f2086f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:08.256328 6708 raft_consensus.cc:1261] T ec2d3ccf0937430f947f3e2bde0506c5 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Refusing update from remote peer 8b8b3ad3aa8f4053b2167dd550f2086f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:59:08.256886 6168 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 0bc706e85ce74cef97a4827ea4d4aa65: Sending DeleteTablet for 3 replicas of tablet 460b4d38c63a4a4ca177a37be70d927f
I20240501 07:59:08.257316 9373 consensus_queue.cc:1035] T ec2d3ccf0937430f947f3e2bde0506c5 P 8b8b3ad3aa8f4053b2167dd550f2086f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:08.257382 6426 tablet_service.cc:1514] Processing DeleteTablet for tablet 460b4d38c63a4a4ca177a37be70d927f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:08 UTC) from {username='slave'} at 127.0.0.1:45034
I20240501 07:59:08.257557 9386 tablet_replica.cc:330] T 460b4d38c63a4a4ca177a37be70d927f P 8315d22c56f7439a9e1fc82f153f018c: stopping tablet replica
I20240501 07:59:08.257722 9386 raft_consensus.cc:2229] T 460b4d38c63a4a4ca177a37be70d927f P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:08.257859 9386 raft_consensus.cc:2258] T 460b4d38c63a4a4ca177a37be70d927f P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:08.257934 6168 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 0bc706e85ce74cef97a4827ea4d4aa65: Sending DeleteTablet for 4 replicas of tablet 7d63a75f2194484f8a59709a209f00ea
I20240501 07:59:08.258036 6687 tablet_service.cc:1514] Processing DeleteTablet for tablet 460b4d38c63a4a4ca177a37be70d927f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:08 UTC) from {username='slave'} at 127.0.0.1:38524
I20240501 07:59:08.258211 9387 tablet_replica.cc:330] T 460b4d38c63a4a4ca177a37be70d927f P 1f9cb72be98941b5a8e504cbe34132b9: stopping tablet replica
I20240501 07:59:08.258343 9387 raft_consensus.cc:2229] T 460b4d38c63a4a4ca177a37be70d927f P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 LEADER]: Raft consensus shutting down.
I20240501 07:59:08.258430 9386 ts_tablet_manager.cc:1875] T 460b4d38c63a4a4ca177a37be70d927f P 8315d22c56f7439a9e1fc82f153f018c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:59:08.258548 9387 raft_consensus.cc:2258] T 460b4d38c63a4a4ca177a37be70d927f P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:08.258942 6168 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 0bc706e85ce74cef97a4827ea4d4aa65: Sending DeleteTablet for 3 replicas of tablet c55dadffc24340338d0439b0ab2855f4
I20240501 07:59:08.259019 9387 ts_tablet_manager.cc:1875] T 460b4d38c63a4a4ca177a37be70d927f P 1f9cb72be98941b5a8e504cbe34132b9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:59:08.259048 8905 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d63a75f2194484f8a59709a209f00ea with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:08 UTC) from {username='slave'} at 127.0.0.1:47202
I20240501 07:59:08.260454 9534 raft_consensus.cc:2932] T ec2d3ccf0937430f947f3e2bde0506c5 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { replace: true } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: true } } }
I20240501 07:59:08.260792 9386 ts_tablet_manager.cc:1888] T 460b4d38c63a4a4ca177a37be70d927f P 8315d22c56f7439a9e1fc82f153f018c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 07:59:08.260859 9386 log.cc:1196] T 460b4d38c63a4a4ca177a37be70d927f P 8315d22c56f7439a9e1fc82f153f018c: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/460b4d38c63a4a4ca177a37be70d927f
I20240501 07:59:08.261051 6294 tablet_service.cc:1514] Processing DeleteTablet for tablet 460b4d38c63a4a4ca177a37be70d927f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:08 UTC) from {username='slave'} at 127.0.0.1:45480
I20240501 07:59:08.261065 6686 tablet_service.cc:1514] Processing DeleteTablet for tablet c55dadffc24340338d0439b0ab2855f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:08 UTC) from {username='slave'} at 127.0.0.1:38524
I20240501 07:59:08.261103 6295 tablet_service.cc:1514] Processing DeleteTablet for tablet c55dadffc24340338d0439b0ab2855f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:08 UTC) from {username='slave'} at 127.0.0.1:45480
I20240501 07:59:08.261247 9388 tablet_replica.cc:330] T 460b4d38c63a4a4ca177a37be70d927f P 8b8b3ad3aa8f4053b2167dd550f2086f: stopping tablet replica
I20240501 07:59:08.261260 6424 tablet_service.cc:1514] Processing DeleteTablet for tablet c55dadffc24340338d0439b0ab2855f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:08 UTC) from {username='slave'} at 127.0.0.1:45034
I20240501 07:59:08.261356 6687 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d63a75f2194484f8a59709a209f00ea with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:08 UTC) from {username='slave'} at 127.0.0.1:38524
I20240501 07:59:08.261390 9388 raft_consensus.cc:2229] T 460b4d38c63a4a4ca177a37be70d927f P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:08.261400 6426 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d63a75f2194484f8a59709a209f00ea with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:08 UTC) from {username='slave'} at 127.0.0.1:45034
I20240501 07:59:08.261533 9388 raft_consensus.cc:2258] T 460b4d38c63a4a4ca177a37be70d927f P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:08.261538 6556 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d63a75f2194484f8a59709a209f00ea with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:08 UTC) from {username='slave'} at 127.0.0.1:53874
I20240501 07:59:08.261699 9385 tablet_replica.cc:330] T 7d63a75f2194484f8a59709a209f00ea P 3b164cbbc00f4e8eb3254ddd349a3ff9: stopping tablet replica
I20240501 07:59:08.261845 9385 raft_consensus.cc:2229] T 7d63a75f2194484f8a59709a209f00ea P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:08.262038 9385 raft_consensus.cc:2258] T 7d63a75f2194484f8a59709a209f00ea P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:08.262043 9388 ts_tablet_manager.cc:1875] T 460b4d38c63a4a4ca177a37be70d927f P 8b8b3ad3aa8f4053b2167dd550f2086f: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240501 07:59:08.262333 6159 catalog_manager.cc:4797] TS 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134:40141): delete failed for tablet 7d63a75f2194484f8a59709a209f00ea because the tablet was not found. No further retry: Not found: Tablet not found: 7d63a75f2194484f8a59709a209f00ea
I20240501 07:59:08.262866 9385 ts_tablet_manager.cc:1875] T 7d63a75f2194484f8a59709a209f00ea P 3b164cbbc00f4e8eb3254ddd349a3ff9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:59:08.264151 9388 ts_tablet_manager.cc:1888] T 460b4d38c63a4a4ca177a37be70d927f P 8b8b3ad3aa8f4053b2167dd550f2086f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 07:59:08.264223 9388 log.cc:1196] T 460b4d38c63a4a4ca177a37be70d927f P 8b8b3ad3aa8f4053b2167dd550f2086f: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/460b4d38c63a4a4ca177a37be70d927f
I20240501 07:59:08.264472 9388 ts_tablet_manager.cc:1909] T 460b4d38c63a4a4ca177a37be70d927f P 8b8b3ad3aa8f4053b2167dd550f2086f: Deleting consensus metadata
I20240501 07:59:08.264933 9385 ts_tablet_manager.cc:1888] T 7d63a75f2194484f8a59709a209f00ea P 3b164cbbc00f4e8eb3254ddd349a3ff9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:59:08.265034 9385 log.cc:1196] T 7d63a75f2194484f8a59709a209f00ea P 3b164cbbc00f4e8eb3254ddd349a3ff9: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/7d63a75f2194484f8a59709a209f00ea
I20240501 07:59:08.265275 9385 ts_tablet_manager.cc:1909] T 7d63a75f2194484f8a59709a209f00ea P 3b164cbbc00f4e8eb3254ddd349a3ff9: Deleting consensus metadata
I20240501 07:59:08.265883 6168 catalog_manager.cc:5484] T ec2d3ccf0937430f947f3e2bde0506c5 P 8b8b3ad3aa8f4053b2167dd550f2086f reported cstate change: config changed from index -1 to 4, NON_VOTER 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 07:59:08.266220 6575 raft_consensus.cc:2932] T ec2d3ccf0937430f947f3e2bde0506c5 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { replace: true } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: true } } }
I20240501 07:59:08.266495 9386 ts_tablet_manager.cc:1909] T 460b4d38c63a4a4ca177a37be70d927f P 8315d22c56f7439a9e1fc82f153f018c: Deleting consensus metadata
I20240501 07:59:08.267129 9386 tablet_replica.cc:330] T c55dadffc24340338d0439b0ab2855f4 P 8315d22c56f7439a9e1fc82f153f018c: stopping tablet replica
I20240501 07:59:08.267228 6159 catalog_manager.cc:4830] TS 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130:32941): tablet 460b4d38c63a4a4ca177a37be70d927f (table rebalance_test_table_2 [id=a8027b31a8b34320800f512d75105639]) successfully deleted
I20240501 07:59:08.267259 9386 raft_consensus.cc:2229] T c55dadffc24340338d0439b0ab2855f4 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:08.267350 9386 raft_consensus.cc:2258] T c55dadffc24340338d0439b0ab2855f4 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:08.267776 9386 ts_tablet_manager.cc:1875] T c55dadffc24340338d0439b0ab2855f4 P 8315d22c56f7439a9e1fc82f153f018c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:59:08.269680 6708 raft_consensus.cc:2932] T ec2d3ccf0937430f947f3e2bde0506c5 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { replace: true } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: true } } }
I20240501 07:59:08.270606 6160 catalog_manager.cc:4830] TS 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131:34471): tablet 7d63a75f2194484f8a59709a209f00ea (table rebalance_test_table_2 [id=a8027b31a8b34320800f512d75105639]) successfully deleted
I20240501 07:59:08.271185 9386 ts_tablet_manager.cc:1888] T c55dadffc24340338d0439b0ab2855f4 P 8315d22c56f7439a9e1fc82f153f018c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 07:59:08.271262 9386 log.cc:1196] T c55dadffc24340338d0439b0ab2855f4 P 8315d22c56f7439a9e1fc82f153f018c: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/c55dadffc24340338d0439b0ab2855f4
I20240501 07:59:08.271533 9386 ts_tablet_manager.cc:1909] T c55dadffc24340338d0439b0ab2855f4 P 8315d22c56f7439a9e1fc82f153f018c: Deleting consensus metadata
I20240501 07:59:08.271684 9387 ts_tablet_manager.cc:1888] T 460b4d38c63a4a4ca177a37be70d927f P 1f9cb72be98941b5a8e504cbe34132b9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 07:59:08.271762 9387 log.cc:1196] T 460b4d38c63a4a4ca177a37be70d927f P 1f9cb72be98941b5a8e504cbe34132b9: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/460b4d38c63a4a4ca177a37be70d927f
I20240501 07:59:08.272007 9386 tablet_replica.cc:330] T 7d63a75f2194484f8a59709a209f00ea P 8315d22c56f7439a9e1fc82f153f018c: stopping tablet replica
I20240501 07:59:08.272017 9387 ts_tablet_manager.cc:1909] T 460b4d38c63a4a4ca177a37be70d927f P 1f9cb72be98941b5a8e504cbe34132b9: Deleting consensus metadata
I20240501 07:59:08.272122 9386 raft_consensus.cc:2229] T 7d63a75f2194484f8a59709a209f00ea P 8315d22c56f7439a9e1fc82f153f018c [term 1 LEADER]: Raft consensus shutting down.
I20240501 07:59:08.272316 9386 raft_consensus.cc:2258] T 7d63a75f2194484f8a59709a209f00ea P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:08.272547 9387 tablet_replica.cc:330] T c55dadffc24340338d0439b0ab2855f4 P 1f9cb72be98941b5a8e504cbe34132b9: stopping tablet replica
I20240501 07:59:08.272634 6159 catalog_manager.cc:4830] TS 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132:36773): tablet 460b4d38c63a4a4ca177a37be70d927f (table rebalance_test_table_2 [id=a8027b31a8b34320800f512d75105639]) successfully deleted
I20240501 07:59:08.272691 9387 raft_consensus.cc:2229] T c55dadffc24340338d0439b0ab2855f4 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:08.272765 6159 catalog_manager.cc:4830] TS 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130:32941): tablet c55dadffc24340338d0439b0ab2855f4 (table rebalance_test_table_2 [id=a8027b31a8b34320800f512d75105639]) successfully deleted
I20240501 07:59:08.272799 9387 raft_consensus.cc:2258] T c55dadffc24340338d0439b0ab2855f4 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:08.273355 9387 ts_tablet_manager.cc:1875] T c55dadffc24340338d0439b0ab2855f4 P 1f9cb72be98941b5a8e504cbe34132b9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:59:08.273741 9388 tablet_replica.cc:330] T c55dadffc24340338d0439b0ab2855f4 P 8b8b3ad3aa8f4053b2167dd550f2086f: stopping tablet replica
I20240501 07:59:08.273906 9388 raft_consensus.cc:2229] T c55dadffc24340338d0439b0ab2855f4 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 LEADER]: Raft consensus shutting down.
I20240501 07:59:08.274145 9386 ts_tablet_manager.cc:1875] T 7d63a75f2194484f8a59709a209f00ea P 8315d22c56f7439a9e1fc82f153f018c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:59:08.274118 9388 raft_consensus.cc:2258] T c55dadffc24340338d0439b0ab2855f4 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:08.274724 9388 ts_tablet_manager.cc:1875] T c55dadffc24340338d0439b0ab2855f4 P 8b8b3ad3aa8f4053b2167dd550f2086f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:59:08.275426 6157 catalog_manager.cc:4830] TS 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129:46875): tablet 460b4d38c63a4a4ca177a37be70d927f (table rebalance_test_table_2 [id=a8027b31a8b34320800f512d75105639]) successfully deleted
I20240501 07:59:08.275550 9387 ts_tablet_manager.cc:1888] T c55dadffc24340338d0439b0ab2855f4 P 1f9cb72be98941b5a8e504cbe34132b9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 07:59:08.275616 9387 log.cc:1196] T c55dadffc24340338d0439b0ab2855f4 P 1f9cb72be98941b5a8e504cbe34132b9: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/c55dadffc24340338d0439b0ab2855f4
I20240501 07:59:08.275897 9387 ts_tablet_manager.cc:1909] T c55dadffc24340338d0439b0ab2855f4 P 1f9cb72be98941b5a8e504cbe34132b9: Deleting consensus metadata
I20240501 07:59:08.276206 9386 ts_tablet_manager.cc:1888] T 7d63a75f2194484f8a59709a209f00ea P 8315d22c56f7439a9e1fc82f153f018c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:59:08.276809 9387 tablet_replica.cc:330] T 7d63a75f2194484f8a59709a209f00ea P 1f9cb72be98941b5a8e504cbe34132b9: stopping tablet replica
I20240501 07:59:08.276866 6159 catalog_manager.cc:4830] TS 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132:36773): tablet c55dadffc24340338d0439b0ab2855f4 (table rebalance_test_table_2 [id=a8027b31a8b34320800f512d75105639]) successfully deleted
I20240501 07:59:08.277097 9387 raft_consensus.cc:2229] T 7d63a75f2194484f8a59709a209f00ea P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:08.277382 9386 log.cc:1196] T 7d63a75f2194484f8a59709a209f00ea P 8315d22c56f7439a9e1fc82f153f018c: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/7d63a75f2194484f8a59709a209f00ea
I20240501 07:59:08.277666 9386 ts_tablet_manager.cc:1909] T 7d63a75f2194484f8a59709a209f00ea P 8315d22c56f7439a9e1fc82f153f018c: Deleting consensus metadata
I20240501 07:59:08.277772 9387 raft_consensus.cc:2258] T 7d63a75f2194484f8a59709a209f00ea P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:08.278321 9388 ts_tablet_manager.cc:1888] T c55dadffc24340338d0439b0ab2855f4 P 8b8b3ad3aa8f4053b2167dd550f2086f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 07:59:08.278457 9387 ts_tablet_manager.cc:1875] T 7d63a75f2194484f8a59709a209f00ea P 1f9cb72be98941b5a8e504cbe34132b9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:59:08.278457 6159 catalog_manager.cc:4830] TS 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130:32941): tablet 7d63a75f2194484f8a59709a209f00ea (table rebalance_test_table_2 [id=a8027b31a8b34320800f512d75105639]) successfully deleted
I20240501 07:59:08.278678 9388 log.cc:1196] T c55dadffc24340338d0439b0ab2855f4 P 8b8b3ad3aa8f4053b2167dd550f2086f: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/c55dadffc24340338d0439b0ab2855f4
I20240501 07:59:08.278944 9388 ts_tablet_manager.cc:1909] T c55dadffc24340338d0439b0ab2855f4 P 8b8b3ad3aa8f4053b2167dd550f2086f: Deleting consensus metadata
I20240501 07:59:08.279733 6157 catalog_manager.cc:4830] TS 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129:46875): tablet c55dadffc24340338d0439b0ab2855f4 (table rebalance_test_table_2 [id=a8027b31a8b34320800f512d75105639]) successfully deleted
I20240501 07:59:08.280316 9387 ts_tablet_manager.cc:1888] T 7d63a75f2194484f8a59709a209f00ea P 1f9cb72be98941b5a8e504cbe34132b9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:59:08.280375 9387 log.cc:1196] T 7d63a75f2194484f8a59709a209f00ea P 1f9cb72be98941b5a8e504cbe34132b9: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/7d63a75f2194484f8a59709a209f00ea
I20240501 07:59:08.280577 9387 ts_tablet_manager.cc:1909] T 7d63a75f2194484f8a59709a209f00ea P 1f9cb72be98941b5a8e504cbe34132b9: Deleting consensus metadata
I20240501 07:59:08.281255 6159 catalog_manager.cc:4830] TS 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132:36773): tablet 7d63a75f2194484f8a59709a209f00ea (table rebalance_test_table_2 [id=a8027b31a8b34320800f512d75105639]) successfully deleted
W20240501 07:59:08.282202 6250 consensus_peers.cc:485] T ec2d3ccf0937430f947f3e2bde0506c5 P 8b8b3ad3aa8f4053b2167dd550f2086f -> Peer 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134:40141): Couldn't send request to peer 13d05ebdca8c4f489c597ce550dd773d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ec2d3ccf0937430f947f3e2bde0506c5. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:08.286227 6708 consensus_queue.cc:237] T 404bb81b9c404dd288d2f8160a4273da P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: true } }
I20240501 07:59:08.287715 6444 raft_consensus.cc:1261] T 404bb81b9c404dd288d2f8160a4273da P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Refusing update from remote peer 1f9cb72be98941b5a8e504cbe34132b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:59:08.287817 6575 raft_consensus.cc:1261] T 404bb81b9c404dd288d2f8160a4273da P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Refusing update from remote peer 1f9cb72be98941b5a8e504cbe34132b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:59:08.288208 9479 consensus_queue.cc:1035] T 404bb81b9c404dd288d2f8160a4273da P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:08.288404 9398 consensus_queue.cc:1035] T 404bb81b9c404dd288d2f8160a4273da P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:08.290136 9479 raft_consensus.cc:2932] T 404bb81b9c404dd288d2f8160a4273da P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: true } } }
I20240501 07:59:08.290571 6575 raft_consensus.cc:2932] T 404bb81b9c404dd288d2f8160a4273da P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: true } } }
W20240501 07:59:08.291668 6642 consensus_peers.cc:485] T 404bb81b9c404dd288d2f8160a4273da P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 404bb81b9c404dd288d2f8160a4273da. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:08.291620 6444 raft_consensus.cc:2932] T 404bb81b9c404dd288d2f8160a4273da P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: true } } }
I20240501 07:59:08.294209 6168 catalog_manager.cc:5484] T 404bb81b9c404dd288d2f8160a4273da P 1f9cb72be98941b5a8e504cbe34132b9 reported cstate change: config changed from index -1 to 4, NON_VOTER 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "1f9cb72be98941b5a8e504cbe34132b9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 07:59:08.300774 6708 consensus_queue.cc:237] T 3b9b71d44f754353b2321e8c36dd4dfd P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: true } }
I20240501 07:59:08.302181 6575 raft_consensus.cc:1261] T 3b9b71d44f754353b2321e8c36dd4dfd P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Refusing update from remote peer 1f9cb72be98941b5a8e504cbe34132b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:59:08.302455 6313 raft_consensus.cc:1261] T 3b9b71d44f754353b2321e8c36dd4dfd P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Refusing update from remote peer 1f9cb72be98941b5a8e504cbe34132b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:59:08.302547 9398 consensus_queue.cc:1035] T 3b9b71d44f754353b2321e8c36dd4dfd P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 07:59:08.302809 6642 consensus_peers.cc:485] T 3b9b71d44f754353b2321e8c36dd4dfd P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3b9b71d44f754353b2321e8c36dd4dfd. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:08.302923 9398 consensus_queue.cc:1035] T 3b9b71d44f754353b2321e8c36dd4dfd P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:08.304237 8710 raft_consensus.cc:2932] T 3b9b71d44f754353b2321e8c36dd4dfd P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: true } } }
I20240501 07:59:08.304620 6575 raft_consensus.cc:2932] T 3b9b71d44f754353b2321e8c36dd4dfd P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: true } } }
I20240501 07:59:08.305138 6313 raft_consensus.cc:2932] T 3b9b71d44f754353b2321e8c36dd4dfd P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: true } } }
I20240501 07:59:08.307101 6172 catalog_manager.cc:5484] T 3b9b71d44f754353b2321e8c36dd4dfd P 1f9cb72be98941b5a8e504cbe34132b9 reported cstate change: config changed from index -1 to 4, NON_VOTER a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "1f9cb72be98941b5a8e504cbe34132b9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 07:59:08.315722 6575 consensus_queue.cc:237] T ff9dc384dbac47dc9795617c64688943 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: true } }
I20240501 07:59:08.317572 6708 raft_consensus.cc:1261] T ff9dc384dbac47dc9795617c64688943 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Refusing update from remote peer 3b164cbbc00f4e8eb3254ddd349a3ff9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:59:08.317965 8680 consensus_queue.cc:1035] T ff9dc384dbac47dc9795617c64688943 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:08.318718 6313 raft_consensus.cc:1261] T ff9dc384dbac47dc9795617c64688943 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Refusing update from remote peer 3b164cbbc00f4e8eb3254ddd349a3ff9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:59:08.319408 8680 consensus_queue.cc:1035] T ff9dc384dbac47dc9795617c64688943 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:08.319835 8680 raft_consensus.cc:2932] T ff9dc384dbac47dc9795617c64688943 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: true } } }
I20240501 07:59:08.320350 6708 raft_consensus.cc:2932] T ff9dc384dbac47dc9795617c64688943 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: true } } }
I20240501 07:59:08.323293 6314 raft_consensus.cc:2932] T ff9dc384dbac47dc9795617c64688943 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: true } } }
I20240501 07:59:08.323849 6166 catalog_manager.cc:5484] T ff9dc384dbac47dc9795617c64688943 P 1f9cb72be98941b5a8e504cbe34132b9 reported cstate change: config changed from index -1 to 4, NON_VOTER 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: true } } }
W20240501 07:59:08.324432 6512 consensus_peers.cc:485] T ff9dc384dbac47dc9795617c64688943 P 3b164cbbc00f4e8eb3254ddd349a3ff9 -> Peer 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134:40141): Couldn't send request to peer 13d05ebdca8c4f489c597ce550dd773d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ff9dc384dbac47dc9795617c64688943. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:08.332567 6314 consensus_queue.cc:237] T 5ee738571af740c1b494d4f17413d1d8 P 8b8b3ad3aa8f4053b2167dd550f2086f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { replace: true } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: true } }
I20240501 07:59:08.334275 6577 raft_consensus.cc:1261] T 5ee738571af740c1b494d4f17413d1d8 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Refusing update from remote peer 8b8b3ad3aa8f4053b2167dd550f2086f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:59:08.334851 9373 consensus_queue.cc:1035] T 5ee738571af740c1b494d4f17413d1d8 P 8b8b3ad3aa8f4053b2167dd550f2086f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:08.335405 6708 raft_consensus.cc:1261] T 5ee738571af740c1b494d4f17413d1d8 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Refusing update from remote peer 8b8b3ad3aa8f4053b2167dd550f2086f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:59:08.335969 9373 consensus_queue.cc:1035] T 5ee738571af740c1b494d4f17413d1d8 P 8b8b3ad3aa8f4053b2167dd550f2086f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:08.337162 9373 raft_consensus.cc:2932] T 5ee738571af740c1b494d4f17413d1d8 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { replace: true } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: true } } }
I20240501 07:59:08.337638 6577 raft_consensus.cc:2932] T 5ee738571af740c1b494d4f17413d1d8 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { replace: true } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: true } } }
W20240501 07:59:08.338008 6248 consensus_peers.cc:485] T 5ee738571af740c1b494d4f17413d1d8 P 8b8b3ad3aa8f4053b2167dd550f2086f -> Peer 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133:39859): Couldn't send request to peer 74077b61d6f54342bf9736b4a9f2eade. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5ee738571af740c1b494d4f17413d1d8. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:08.339962 6708 raft_consensus.cc:2932] T 5ee738571af740c1b494d4f17413d1d8 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { replace: true } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: true } } }
I20240501 07:59:08.340615 6168 catalog_manager.cc:5484] T 5ee738571af740c1b494d4f17413d1d8 P 8b8b3ad3aa8f4053b2167dd550f2086f reported cstate change: config changed from index -1 to 4, NON_VOTER 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 07:59:08.349098 6577 consensus_queue.cc:237] T 6e2ac8c8857d45b88d3b51147d2609aa P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: true } }
I20240501 07:59:08.350520 6314 raft_consensus.cc:1261] T 6e2ac8c8857d45b88d3b51147d2609aa P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Refusing update from remote peer 3b164cbbc00f4e8eb3254ddd349a3ff9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:59:08.350647 6444 raft_consensus.cc:1261] T 6e2ac8c8857d45b88d3b51147d2609aa P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Refusing update from remote peer 3b164cbbc00f4e8eb3254ddd349a3ff9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 07:59:08.350785 6511 consensus_peers.cc:485] T 6e2ac8c8857d45b88d3b51147d2609aa P 3b164cbbc00f4e8eb3254ddd349a3ff9 -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6e2ac8c8857d45b88d3b51147d2609aa. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:08.350909 8166 consensus_queue.cc:1035] T 6e2ac8c8857d45b88d3b51147d2609aa P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:08.351329 8166 consensus_queue.cc:1035] T 6e2ac8c8857d45b88d3b51147d2609aa P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:08.352803 9507 raft_consensus.cc:2932] T 6e2ac8c8857d45b88d3b51147d2609aa P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: true } } }
I20240501 07:59:08.353370 6314 raft_consensus.cc:2932] T 6e2ac8c8857d45b88d3b51147d2609aa P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: true } } }
I20240501 07:59:08.354688 9400 consensus_queue.cc:1035] T fb6b46072a8d46e18b53b9f8de702859 P 13d05ebdca8c4f489c597ce550dd773d [LEADER]: Connected to new peer: Peer: permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:08.355273 6444 raft_consensus.cc:2932] T 6e2ac8c8857d45b88d3b51147d2609aa P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: true } } }
I20240501 07:59:08.355984 6166 catalog_manager.cc:5484] T 6e2ac8c8857d45b88d3b51147d2609aa P 3b164cbbc00f4e8eb3254ddd349a3ff9 reported cstate change: config changed from index -1 to 4, NON_VOTER 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 07:59:08.362020 9581 consensus_queue.cc:1035] T fb6b46072a8d46e18b53b9f8de702859 P 13d05ebdca8c4f489c597ce550dd773d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:08.369931 6708 consensus_queue.cc:237] T 187940ba79f546b1ad79e7ec1de76442 P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { replace: true } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: true } }
I20240501 07:59:08.371798 6314 raft_consensus.cc:1261] T 187940ba79f546b1ad79e7ec1de76442 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Refusing update from remote peer 1f9cb72be98941b5a8e504cbe34132b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:59:08.372165 8710 consensus_queue.cc:1035] T 187940ba79f546b1ad79e7ec1de76442 P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:08.372814 6577 raft_consensus.cc:1261] T 187940ba79f546b1ad79e7ec1de76442 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Refusing update from remote peer 1f9cb72be98941b5a8e504cbe34132b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:59:08.372754 9406 consensus_queue.cc:1035] T 6292ff0c7e624dd7b24715b83a30fff8 P 74077b61d6f54342bf9736b4a9f2eade [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:08.373437 8710 consensus_queue.cc:1035] T 187940ba79f546b1ad79e7ec1de76442 P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:08.374527 9398 raft_consensus.cc:2932] T 187940ba79f546b1ad79e7ec1de76442 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { replace: true } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: true } } }
I20240501 07:59:08.374979 6314 raft_consensus.cc:2932] T 187940ba79f546b1ad79e7ec1de76442 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { replace: true } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: true } } }
W20240501 07:59:08.378576 6643 consensus_peers.cc:485] T 187940ba79f546b1ad79e7ec1de76442 P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134:40141): Couldn't send request to peer 13d05ebdca8c4f489c597ce550dd773d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 187940ba79f546b1ad79e7ec1de76442. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:08.378439 6172 catalog_manager.cc:5484] T 187940ba79f546b1ad79e7ec1de76442 P 8b8b3ad3aa8f4053b2167dd550f2086f reported cstate change: config changed from index -1 to 4, NON_VOTER 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "1f9cb72be98941b5a8e504cbe34132b9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { replace: true } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: true } } }
I20240501 07:59:08.379752 6575 raft_consensus.cc:2932] T 187940ba79f546b1ad79e7ec1de76442 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { replace: true } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: true } } }
I20240501 07:59:08.386585 9406 consensus_queue.cc:1035] T 6292ff0c7e624dd7b24715b83a30fff8 P 74077b61d6f54342bf9736b4a9f2eade [LEADER]: Connected to new peer: Peer: permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:08.390565 9258 raft_consensus.cc:482] T 4cdabbbfd1df4a5a9683e3c8e569d624 P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:08.390678 9258 raft_consensus.cc:504] T 4cdabbbfd1df4a5a9683e3c8e569d624 P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } }
I20240501 07:59:08.391880 9258 leader_election.cc:288] T 4cdabbbfd1df4a5a9683e3c8e569d624 P 83f20e84af4c40d09a4804b6b26e697f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133:39859), 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134:40141)
I20240501 07:59:08.393045 6444 consensus_queue.cc:237] T 782e37a4ebb34a68b461a74115ae8703 P 8315d22c56f7439a9e1fc82f153f018c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: true } }
I20240501 07:59:08.395555 6575 raft_consensus.cc:1261] T 782e37a4ebb34a68b461a74115ae8703 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Refusing update from remote peer 8315d22c56f7439a9e1fc82f153f018c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:59:08.396085 9527 consensus_queue.cc:1035] T 782e37a4ebb34a68b461a74115ae8703 P 8315d22c56f7439a9e1fc82f153f018c [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:08.397011 6314 raft_consensus.cc:1261] T 782e37a4ebb34a68b461a74115ae8703 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Refusing update from remote peer 8315d22c56f7439a9e1fc82f153f018c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:59:08.397483 9527 consensus_queue.cc:1035] T 782e37a4ebb34a68b461a74115ae8703 P 8315d22c56f7439a9e1fc82f153f018c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:08.397485 8794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cdabbbfd1df4a5a9683e3c8e569d624" candidate_uuid: "83f20e84af4c40d09a4804b6b26e697f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74077b61d6f54342bf9736b4a9f2eade" is_pre_election: true
I20240501 07:59:08.397639 8794 raft_consensus.cc:2454] T 4cdabbbfd1df4a5a9683e3c8e569d624 P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83f20e84af4c40d09a4804b6b26e697f in term 0.
I20240501 07:59:08.398025 9122 leader_election.cc:302] T 4cdabbbfd1df4a5a9683e3c8e569d624 P 83f20e84af4c40d09a4804b6b26e697f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74077b61d6f54342bf9736b4a9f2eade, 83f20e84af4c40d09a4804b6b26e697f; no voters:
I20240501 07:59:08.398279 9258 raft_consensus.cc:2781] T 4cdabbbfd1df4a5a9683e3c8e569d624 P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:08.398347 9258 raft_consensus.cc:482] T 4cdabbbfd1df4a5a9683e3c8e569d624 P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:08.398388 9258 raft_consensus.cc:3037] T 4cdabbbfd1df4a5a9683e3c8e569d624 P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:08.399518 9258 raft_consensus.cc:504] T 4cdabbbfd1df4a5a9683e3c8e569d624 P 83f20e84af4c40d09a4804b6b26e697f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } }
I20240501 07:59:08.399708 9527 raft_consensus.cc:2932] T 782e37a4ebb34a68b461a74115ae8703 P 8315d22c56f7439a9e1fc82f153f018c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: true } } }
I20240501 07:59:08.399907 9258 leader_election.cc:288] T 4cdabbbfd1df4a5a9683e3c8e569d624 P 83f20e84af4c40d09a4804b6b26e697f [CANDIDATE]: Term 1 election: Requested vote from peers 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133:39859), 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134:40141)
W20240501 07:59:08.400305 6380 consensus_peers.cc:485] T 782e37a4ebb34a68b461a74115ae8703 P 8315d22c56f7439a9e1fc82f153f018c -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 782e37a4ebb34a68b461a74115ae8703. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:08.401072 8924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cdabbbfd1df4a5a9683e3c8e569d624" candidate_uuid: "83f20e84af4c40d09a4804b6b26e697f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13d05ebdca8c4f489c597ce550dd773d"
I20240501 07:59:08.401209 8924 raft_consensus.cc:3037] T 4cdabbbfd1df4a5a9683e3c8e569d624 P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:08.401346 8925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cdabbbfd1df4a5a9683e3c8e569d624" candidate_uuid: "83f20e84af4c40d09a4804b6b26e697f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13d05ebdca8c4f489c597ce550dd773d" is_pre_election: true
I20240501 07:59:08.401768 8794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cdabbbfd1df4a5a9683e3c8e569d624" candidate_uuid: "83f20e84af4c40d09a4804b6b26e697f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74077b61d6f54342bf9736b4a9f2eade"
I20240501 07:59:08.401894 8794 raft_consensus.cc:3037] T 4cdabbbfd1df4a5a9683e3c8e569d624 P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:08.402184 8924 raft_consensus.cc:2454] T 4cdabbbfd1df4a5a9683e3c8e569d624 P 13d05ebdca8c4f489c597ce550dd773d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83f20e84af4c40d09a4804b6b26e697f in term 1.
I20240501 07:59:08.402534 9124 leader_election.cc:302] T 4cdabbbfd1df4a5a9683e3c8e569d624 P 83f20e84af4c40d09a4804b6b26e697f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13d05ebdca8c4f489c597ce550dd773d, 83f20e84af4c40d09a4804b6b26e697f; no voters:
I20240501 07:59:08.402711 9258 raft_consensus.cc:2781] T 4cdabbbfd1df4a5a9683e3c8e569d624 P 83f20e84af4c40d09a4804b6b26e697f [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:08.402855 8794 raft_consensus.cc:2454] T 4cdabbbfd1df4a5a9683e3c8e569d624 P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83f20e84af4c40d09a4804b6b26e697f in term 1.
I20240501 07:59:08.402913 9258 raft_consensus.cc:686] T 4cdabbbfd1df4a5a9683e3c8e569d624 P 83f20e84af4c40d09a4804b6b26e697f [term 1 LEADER]: Becoming Leader. State: Replica: 83f20e84af4c40d09a4804b6b26e697f, State: Running, Role: LEADER
I20240501 07:59:08.402918 6575 raft_consensus.cc:2932] T 782e37a4ebb34a68b461a74115ae8703 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: true } } }
I20240501 07:59:08.403158 9258 consensus_queue.cc:237] T 4cdabbbfd1df4a5a9683e3c8e569d624 P 83f20e84af4c40d09a4804b6b26e697f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } }
I20240501 07:59:08.403302 6314 raft_consensus.cc:2932] T 782e37a4ebb34a68b461a74115ae8703 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: true } } }
I20240501 07:59:08.405910 6172 catalog_manager.cc:5484] T 4cdabbbfd1df4a5a9683e3c8e569d624 P 83f20e84af4c40d09a4804b6b26e697f reported cstate change: term changed from 0 to 1, leader changed from <none> to 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136). New cstate: current_term: 1 leader_uuid: "83f20e84af4c40d09a4804b6b26e697f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } health_report { overall_health: UNKNOWN } } }
I20240501 07:59:08.406260 6168 catalog_manager.cc:5484] T 782e37a4ebb34a68b461a74115ae8703 P 8315d22c56f7439a9e1fc82f153f018c reported cstate change: config changed from index -1 to 4, NON_VOTER a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "8315d22c56f7439a9e1fc82f153f018c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 07:59:08.415756 6708 consensus_queue.cc:237] T c3a9036ccfee45498bca9681518d3699 P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { replace: true } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: true } }
I20240501 07:59:08.417378 6444 raft_consensus.cc:1261] T c3a9036ccfee45498bca9681518d3699 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Refusing update from remote peer 1f9cb72be98941b5a8e504cbe34132b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:59:08.417529 6575 raft_consensus.cc:1261] T c3a9036ccfee45498bca9681518d3699 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Refusing update from remote peer 1f9cb72be98941b5a8e504cbe34132b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:59:08.417871 9398 consensus_queue.cc:1035] T c3a9036ccfee45498bca9681518d3699 P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:08.418068 9479 consensus_queue.cc:1035] T c3a9036ccfee45498bca9681518d3699 P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:08.420481 9479 raft_consensus.cc:2932] T c3a9036ccfee45498bca9681518d3699 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { replace: true } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: true } } }
I20240501 07:59:08.420922 6575 raft_consensus.cc:2932] T c3a9036ccfee45498bca9681518d3699 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { replace: true } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: true } } }
W20240501 07:59:08.421510 6641 consensus_peers.cc:485] T c3a9036ccfee45498bca9681518d3699 P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133:39859): Couldn't send request to peer 74077b61d6f54342bf9736b4a9f2eade. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c3a9036ccfee45498bca9681518d3699. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:08.422297 6444 raft_consensus.cc:2932] T c3a9036ccfee45498bca9681518d3699 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { replace: true } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: true } } }
I20240501 07:59:08.424631 6167 catalog_manager.cc:5484] T c3a9036ccfee45498bca9681518d3699 P 3b164cbbc00f4e8eb3254ddd349a3ff9 reported cstate change: config changed from index -1 to 4, NON_VOTER 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "1f9cb72be98941b5a8e504cbe34132b9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { replace: true } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: true } } }
I20240501 07:59:08.433301 6444 consensus_queue.cc:237] T bab34f083c8045e4a6df373db5ebd54b P 8315d22c56f7439a9e1fc82f153f018c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: true } }
I20240501 07:59:08.434782 6314 raft_consensus.cc:1261] T bab34f083c8045e4a6df373db5ebd54b P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Refusing update from remote peer 8315d22c56f7439a9e1fc82f153f018c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 07:59:08.435118 6381 consensus_peers.cc:485] T bab34f083c8045e4a6df373db5ebd54b P 8315d22c56f7439a9e1fc82f153f018c -> Peer 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134:40141): Couldn't send request to peer 13d05ebdca8c4f489c597ce550dd773d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bab34f083c8045e4a6df373db5ebd54b. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:08.435211 6575 raft_consensus.cc:1261] T bab34f083c8045e4a6df373db5ebd54b P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Refusing update from remote peer 8315d22c56f7439a9e1fc82f153f018c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:59:08.435200 9116 consensus_queue.cc:1035] T bab34f083c8045e4a6df373db5ebd54b P 8315d22c56f7439a9e1fc82f153f018c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:08.435602 9116 consensus_queue.cc:1035] T bab34f083c8045e4a6df373db5ebd54b P 8315d22c56f7439a9e1fc82f153f018c [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:08.436900 9489 raft_consensus.cc:2932] T bab34f083c8045e4a6df373db5ebd54b P 8315d22c56f7439a9e1fc82f153f018c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: true } } }
I20240501 07:59:08.437405 6314 raft_consensus.cc:2932] T bab34f083c8045e4a6df373db5ebd54b P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: true } } }
I20240501 07:59:08.437860 6575 raft_consensus.cc:2932] T bab34f083c8045e4a6df373db5ebd54b P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: true } } }
I20240501 07:59:08.440248 6166 catalog_manager.cc:5484] T bab34f083c8045e4a6df373db5ebd54b P 8315d22c56f7439a9e1fc82f153f018c reported cstate change: config changed from index -1 to 4, NON_VOTER 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "8315d22c56f7439a9e1fc82f153f018c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 07:59:08.452435 6708 consensus_queue.cc:237] T 7d51b1940b594c79a7d13d7026b76da8 P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { replace: true } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: true } }
W20240501 07:59:08.454037 6642 consensus_peers.cc:485] T 7d51b1940b594c79a7d13d7026b76da8 P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7d51b1940b594c79a7d13d7026b76da8. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:08.454057 6314 raft_consensus.cc:1261] T 7d51b1940b594c79a7d13d7026b76da8 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Refusing update from remote peer 1f9cb72be98941b5a8e504cbe34132b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:59:08.454283 6575 raft_consensus.cc:1261] T 7d51b1940b594c79a7d13d7026b76da8 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Refusing update from remote peer 1f9cb72be98941b5a8e504cbe34132b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:59:08.454677 9618 consensus_queue.cc:1035] T 7d51b1940b594c79a7d13d7026b76da8 P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:08.454896 8710 consensus_queue.cc:1035] T 7d51b1940b594c79a7d13d7026b76da8 P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:08.456589 9618 raft_consensus.cc:2932] T 7d51b1940b594c79a7d13d7026b76da8 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { replace: true } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: true } } }
I20240501 07:59:08.457094 6575 raft_consensus.cc:2932] T 7d51b1940b594c79a7d13d7026b76da8 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { replace: true } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: true } } }
I20240501 07:59:08.458321 6314 raft_consensus.cc:2932] T 7d51b1940b594c79a7d13d7026b76da8 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { replace: true } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: true } } }
I20240501 07:59:08.460515 6172 catalog_manager.cc:5484] T 7d51b1940b594c79a7d13d7026b76da8 P 1f9cb72be98941b5a8e504cbe34132b9 reported cstate change: config changed from index -1 to 4, NON_VOTER 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "1f9cb72be98941b5a8e504cbe34132b9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 07:59:08.462755 9457 consensus_queue.cc:1035] T f88fe033bce14bf0b1b63a52a7ba137e P a505e7cb133c4392bfda8a4a20874ae9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:08.467109 9255 consensus_queue.cc:1035] T f88fe033bce14bf0b1b63a52a7ba137e P a505e7cb133c4392bfda8a4a20874ae9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:08.475015 6708 consensus_queue.cc:237] T 8454f24b0c28451b8ee51be39ca3e07f P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: true } }
I20240501 07:59:08.476471 6575 raft_consensus.cc:1261] T 8454f24b0c28451b8ee51be39ca3e07f P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Refusing update from remote peer 1f9cb72be98941b5a8e504cbe34132b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:59:08.476531 6314 raft_consensus.cc:1261] T 8454f24b0c28451b8ee51be39ca3e07f P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Refusing update from remote peer 1f9cb72be98941b5a8e504cbe34132b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:59:08.476909 9618 consensus_queue.cc:1035] T 8454f24b0c28451b8ee51be39ca3e07f P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 07:59:08.477116 6642 consensus_peers.cc:485] T 8454f24b0c28451b8ee51be39ca3e07f P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8454f24b0c28451b8ee51be39ca3e07f. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:08.477280 8710 consensus_queue.cc:1035] T 8454f24b0c28451b8ee51be39ca3e07f P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:08.479110 9398 raft_consensus.cc:2932] T 8454f24b0c28451b8ee51be39ca3e07f P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: true } } }
I20240501 07:59:08.479552 6575 raft_consensus.cc:2932] T 8454f24b0c28451b8ee51be39ca3e07f P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: true } } }
I20240501 07:59:08.481035 6314 raft_consensus.cc:2932] T 8454f24b0c28451b8ee51be39ca3e07f P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: true } } }
I20240501 07:59:08.483253 6167 catalog_manager.cc:5484] T 8454f24b0c28451b8ee51be39ca3e07f P 1f9cb72be98941b5a8e504cbe34132b9 reported cstate change: config changed from index -1 to 4, NON_VOTER a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "1f9cb72be98941b5a8e504cbe34132b9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 07:59:08.486033 6168 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55950:
name: "rebalancer_extra_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 07:59:08.493333 8774 tablet_service.cc:1467] Processing CreateTablet for tablet c5a876dfb598428ab350593129505504 (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=5e1db4e06c5e470a9d2f8a20dd23c585]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:08.493686 8774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5a876dfb598428ab350593129505504. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:08.493710 6687 tablet_service.cc:1467] Processing CreateTablet for tablet 54b3efabad7f4b608cc9d931546b5743 (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=5e1db4e06c5e470a9d2f8a20dd23c585]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:08.493902 8905 tablet_service.cc:1467] Processing CreateTablet for tablet cfcc814f4d2d488ba745ecaf0cc6d61a (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=5e1db4e06c5e470a9d2f8a20dd23c585]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:08.493986 6687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54b3efabad7f4b608cc9d931546b5743. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:08.494048 9036 tablet_service.cc:1467] Processing CreateTablet for tablet cfcc814f4d2d488ba745ecaf0cc6d61a (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=5e1db4e06c5e470a9d2f8a20dd23c585]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:08.494153 8905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cfcc814f4d2d488ba745ecaf0cc6d61a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:08.494294 9036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cfcc814f4d2d488ba745ecaf0cc6d61a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:08.496160 6426 tablet_service.cc:1467] Processing CreateTablet for tablet cfcc814f4d2d488ba745ecaf0cc6d61a (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=5e1db4e06c5e470a9d2f8a20dd23c585]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:08.496232 9393 tablet_bootstrap.cc:492] T cfcc814f4d2d488ba745ecaf0cc6d61a P 13d05ebdca8c4f489c597ce550dd773d: Bootstrap starting.
I20240501 07:59:08.496448 6426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cfcc814f4d2d488ba745ecaf0cc6d61a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:08.497187 9393 tablet_bootstrap.cc:654] T cfcc814f4d2d488ba745ecaf0cc6d61a P 13d05ebdca8c4f489c597ce550dd773d: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:08.497298 9169 tablet_service.cc:1467] Processing CreateTablet for tablet c5a876dfb598428ab350593129505504 (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=5e1db4e06c5e470a9d2f8a20dd23c585]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:08.497563 9169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5a876dfb598428ab350593129505504. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:08.497884 9167 tablet_service.cc:1467] Processing CreateTablet for tablet 54b3efabad7f4b608cc9d931546b5743 (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=5e1db4e06c5e470a9d2f8a20dd23c585]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:08.498095 9167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54b3efabad7f4b608cc9d931546b5743. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:08.498925 6424 tablet_service.cc:1467] Processing CreateTablet for tablet c5a876dfb598428ab350593129505504 (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=5e1db4e06c5e470a9d2f8a20dd23c585]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:08.499163 6424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5a876dfb598428ab350593129505504. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:08.499642 9393 tablet_bootstrap.cc:492] T cfcc814f4d2d488ba745ecaf0cc6d61a P 13d05ebdca8c4f489c597ce550dd773d: No bootstrap required, opened a new log
I20240501 07:59:08.499725 9393 ts_tablet_manager.cc:1367] T cfcc814f4d2d488ba745ecaf0cc6d61a P 13d05ebdca8c4f489c597ce550dd773d: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240501 07:59:08.499928 9256 tablet_bootstrap.cc:492] T c5a876dfb598428ab350593129505504 P 83f20e84af4c40d09a4804b6b26e697f: Bootstrap starting.
I20240501 07:59:08.500137 9393 raft_consensus.cc:348] T cfcc814f4d2d488ba745ecaf0cc6d61a P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } }
I20240501 07:59:08.500241 9393 raft_consensus.cc:374] T cfcc814f4d2d488ba745ecaf0cc6d61a P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:08.500272 9393 raft_consensus.cc:729] T cfcc814f4d2d488ba745ecaf0cc6d61a P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13d05ebdca8c4f489c597ce550dd773d, State: Initialized, Role: FOLLOWER
I20240501 07:59:08.500319 9654 tablet_bootstrap.cc:492] T cfcc814f4d2d488ba745ecaf0cc6d61a P 8315d22c56f7439a9e1fc82f153f018c: Bootstrap starting.
I20240501 07:59:08.500414 9393 consensus_queue.cc:260] T cfcc814f4d2d488ba745ecaf0cc6d61a P 13d05ebdca8c4f489c597ce550dd773d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } }
I20240501 07:59:08.500685 9393 ts_tablet_manager.cc:1398] T cfcc814f4d2d488ba745ecaf0cc6d61a P 13d05ebdca8c4f489c597ce550dd773d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:08.500922 9256 tablet_bootstrap.cc:654] T c5a876dfb598428ab350593129505504 P 83f20e84af4c40d09a4804b6b26e697f: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:08.501045 9652 tablet_bootstrap.cc:492] T cfcc814f4d2d488ba745ecaf0cc6d61a P a505e7cb133c4392bfda8a4a20874ae9: Bootstrap starting.
I20240501 07:59:08.501305 6425 tablet_service.cc:1467] Processing CreateTablet for tablet 54b3efabad7f4b608cc9d931546b5743 (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=5e1db4e06c5e470a9d2f8a20dd23c585]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:08.501530 6425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54b3efabad7f4b608cc9d931546b5743. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:08.502241 9652 tablet_bootstrap.cc:654] T cfcc814f4d2d488ba745ecaf0cc6d61a P a505e7cb133c4392bfda8a4a20874ae9: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:08.503269 9652 tablet_bootstrap.cc:492] T cfcc814f4d2d488ba745ecaf0cc6d61a P a505e7cb133c4392bfda8a4a20874ae9: No bootstrap required, opened a new log
I20240501 07:59:08.503388 9652 ts_tablet_manager.cc:1367] T cfcc814f4d2d488ba745ecaf0cc6d61a P a505e7cb133c4392bfda8a4a20874ae9: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 07:59:08.503661 9392 tablet_bootstrap.cc:492] T c5a876dfb598428ab350593129505504 P 74077b61d6f54342bf9736b4a9f2eade: Bootstrap starting.
I20240501 07:59:08.503875 9652 raft_consensus.cc:348] T cfcc814f4d2d488ba745ecaf0cc6d61a P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } }
I20240501 07:59:08.504000 9652 raft_consensus.cc:374] T cfcc814f4d2d488ba745ecaf0cc6d61a P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:08.504040 9652 raft_consensus.cc:729] T cfcc814f4d2d488ba745ecaf0cc6d61a P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a505e7cb133c4392bfda8a4a20874ae9, State: Initialized, Role: FOLLOWER
I20240501 07:59:08.504171 9652 consensus_queue.cc:260] T cfcc814f4d2d488ba745ecaf0cc6d61a P a505e7cb133c4392bfda8a4a20874ae9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } }
I20240501 07:59:08.504451 9652 ts_tablet_manager.cc:1398] T cfcc814f4d2d488ba745ecaf0cc6d61a P a505e7cb133c4392bfda8a4a20874ae9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:08.504572 9392 tablet_bootstrap.cc:654] T c5a876dfb598428ab350593129505504 P 74077b61d6f54342bf9736b4a9f2eade: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:08.504940 9654 tablet_bootstrap.cc:654] T cfcc814f4d2d488ba745ecaf0cc6d61a P 8315d22c56f7439a9e1fc82f153f018c: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:08.505364 9256 tablet_bootstrap.cc:492] T c5a876dfb598428ab350593129505504 P 83f20e84af4c40d09a4804b6b26e697f: No bootstrap required, opened a new log
I20240501 07:59:08.505460 9256 ts_tablet_manager.cc:1367] T c5a876dfb598428ab350593129505504 P 83f20e84af4c40d09a4804b6b26e697f: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 07:59:08.506002 9256 raft_consensus.cc:348] T c5a876dfb598428ab350593129505504 P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } }
I20240501 07:59:08.506108 9256 raft_consensus.cc:374] T c5a876dfb598428ab350593129505504 P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:08.506158 9256 raft_consensus.cc:729] T c5a876dfb598428ab350593129505504 P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83f20e84af4c40d09a4804b6b26e697f, State: Initialized, Role: FOLLOWER
I20240501 07:59:08.506279 9256 consensus_queue.cc:260] T c5a876dfb598428ab350593129505504 P 83f20e84af4c40d09a4804b6b26e697f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } }
I20240501 07:59:08.506523 9256 ts_tablet_manager.cc:1398] T c5a876dfb598428ab350593129505504 P 83f20e84af4c40d09a4804b6b26e697f: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 07:59:08.506687 9256 tablet_bootstrap.cc:492] T 54b3efabad7f4b608cc9d931546b5743 P 83f20e84af4c40d09a4804b6b26e697f: Bootstrap starting.
I20240501 07:59:08.507560 9256 tablet_bootstrap.cc:654] T 54b3efabad7f4b608cc9d931546b5743 P 83f20e84af4c40d09a4804b6b26e697f: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:08.506214 9654 tablet_bootstrap.cc:492] T cfcc814f4d2d488ba745ecaf0cc6d61a P 8315d22c56f7439a9e1fc82f153f018c: No bootstrap required, opened a new log
I20240501 07:59:08.508273 9654 ts_tablet_manager.cc:1367] T cfcc814f4d2d488ba745ecaf0cc6d61a P 8315d22c56f7439a9e1fc82f153f018c: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.002s
I20240501 07:59:08.508708 9654 raft_consensus.cc:348] T cfcc814f4d2d488ba745ecaf0cc6d61a P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } }
I20240501 07:59:08.509053 9256 tablet_bootstrap.cc:492] T 54b3efabad7f4b608cc9d931546b5743 P 83f20e84af4c40d09a4804b6b26e697f: No bootstrap required, opened a new log
I20240501 07:59:08.509130 9256 ts_tablet_manager.cc:1367] T 54b3efabad7f4b608cc9d931546b5743 P 83f20e84af4c40d09a4804b6b26e697f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240501 07:59:08.509202 6575 consensus_queue.cc:237] T b7d57b7fd5234f50afbc859bc997f3e1 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { replace: true } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: true } }
I20240501 07:59:08.508805 9654 raft_consensus.cc:374] T cfcc814f4d2d488ba745ecaf0cc6d61a P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:08.509357 9654 raft_consensus.cc:729] T cfcc814f4d2d488ba745ecaf0cc6d61a P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8315d22c56f7439a9e1fc82f153f018c, State: Initialized, Role: FOLLOWER
I20240501 07:59:08.509496 9654 consensus_queue.cc:260] T cfcc814f4d2d488ba745ecaf0cc6d61a P 8315d22c56f7439a9e1fc82f153f018c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } }
I20240501 07:59:08.509586 9256 raft_consensus.cc:348] T 54b3efabad7f4b608cc9d931546b5743 P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } }
I20240501 07:59:08.509682 9256 raft_consensus.cc:374] T 54b3efabad7f4b608cc9d931546b5743 P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:08.509717 9256 raft_consensus.cc:729] T 54b3efabad7f4b608cc9d931546b5743 P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83f20e84af4c40d09a4804b6b26e697f, State: Initialized, Role: FOLLOWER
I20240501 07:59:08.509791 9654 ts_tablet_manager.cc:1398] T cfcc814f4d2d488ba745ecaf0cc6d61a P 8315d22c56f7439a9e1fc82f153f018c: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 07:59:08.509838 9256 consensus_queue.cc:260] T 54b3efabad7f4b608cc9d931546b5743 P 83f20e84af4c40d09a4804b6b26e697f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } }
I20240501 07:59:08.509974 9654 tablet_bootstrap.cc:492] T c5a876dfb598428ab350593129505504 P 8315d22c56f7439a9e1fc82f153f018c: Bootstrap starting.
I20240501 07:59:08.510082 9256 ts_tablet_manager.cc:1398] T 54b3efabad7f4b608cc9d931546b5743 P 83f20e84af4c40d09a4804b6b26e697f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:59:08.510846 9654 tablet_bootstrap.cc:654] T c5a876dfb598428ab350593129505504 P 8315d22c56f7439a9e1fc82f153f018c: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:08.512006 9654 tablet_bootstrap.cc:492] T c5a876dfb598428ab350593129505504 P 8315d22c56f7439a9e1fc82f153f018c: No bootstrap required, opened a new log
I20240501 07:59:08.512027 9392 tablet_bootstrap.cc:492] T c5a876dfb598428ab350593129505504 P 74077b61d6f54342bf9736b4a9f2eade: No bootstrap required, opened a new log
I20240501 07:59:08.512109 9392 ts_tablet_manager.cc:1367] T c5a876dfb598428ab350593129505504 P 74077b61d6f54342bf9736b4a9f2eade: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.001s
I20240501 07:59:08.512142 9654 ts_tablet_manager.cc:1367] T c5a876dfb598428ab350593129505504 P 8315d22c56f7439a9e1fc82f153f018c: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240501 07:59:08.512497 9392 raft_consensus.cc:348] T c5a876dfb598428ab350593129505504 P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } }
I20240501 07:59:08.512598 9392 raft_consensus.cc:374] T c5a876dfb598428ab350593129505504 P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:08.512625 9392 raft_consensus.cc:729] T c5a876dfb598428ab350593129505504 P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74077b61d6f54342bf9736b4a9f2eade, State: Initialized, Role: FOLLOWER
I20240501 07:59:08.512643 6707 raft_consensus.cc:1261] T b7d57b7fd5234f50afbc859bc997f3e1 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Refusing update from remote peer 3b164cbbc00f4e8eb3254ddd349a3ff9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:59:08.512746 9392 consensus_queue.cc:260] T c5a876dfb598428ab350593129505504 P 74077b61d6f54342bf9736b4a9f2eade [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } }
I20240501 07:59:08.513029 9392 ts_tablet_manager.cc:1398] T c5a876dfb598428ab350593129505504 P 74077b61d6f54342bf9736b4a9f2eade: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 07:59:08.513382 6314 raft_consensus.cc:1261] T b7d57b7fd5234f50afbc859bc997f3e1 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Refusing update from remote peer 3b164cbbc00f4e8eb3254ddd349a3ff9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:59:08.513579 8166 consensus_queue.cc:1035] T b7d57b7fd5234f50afbc859bc997f3e1 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:08.513674 9654 raft_consensus.cc:348] T c5a876dfb598428ab350593129505504 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } }
I20240501 07:59:08.513787 9654 raft_consensus.cc:374] T c5a876dfb598428ab350593129505504 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:08.513823 9654 raft_consensus.cc:729] T c5a876dfb598428ab350593129505504 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8315d22c56f7439a9e1fc82f153f018c, State: Initialized, Role: FOLLOWER
I20240501 07:59:08.513938 9654 consensus_queue.cc:260] T c5a876dfb598428ab350593129505504 P 8315d22c56f7439a9e1fc82f153f018c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } }
I20240501 07:59:08.514027 8680 consensus_queue.cc:1035] T b7d57b7fd5234f50afbc859bc997f3e1 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:08.514230 9654 ts_tablet_manager.cc:1398] T c5a876dfb598428ab350593129505504 P 8315d22c56f7439a9e1fc82f153f018c: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 07:59:08.514468 9654 tablet_bootstrap.cc:492] T 54b3efabad7f4b608cc9d931546b5743 P 8315d22c56f7439a9e1fc82f153f018c: Bootstrap starting.
I20240501 07:59:08.515542 9654 tablet_bootstrap.cc:654] T 54b3efabad7f4b608cc9d931546b5743 P 8315d22c56f7439a9e1fc82f153f018c: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:08.517572 9654 tablet_bootstrap.cc:492] T 54b3efabad7f4b608cc9d931546b5743 P 8315d22c56f7439a9e1fc82f153f018c: No bootstrap required, opened a new log
I20240501 07:59:08.517663 9654 ts_tablet_manager.cc:1367] T 54b3efabad7f4b608cc9d931546b5743 P 8315d22c56f7439a9e1fc82f153f018c: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
W20240501 07:59:08.517990 6510 consensus_peers.cc:485] T b7d57b7fd5234f50afbc859bc997f3e1 P 3b164cbbc00f4e8eb3254ddd349a3ff9 -> Peer 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133:39859): Couldn't send request to peer 74077b61d6f54342bf9736b4a9f2eade. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b7d57b7fd5234f50afbc859bc997f3e1. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:08.518126 9654 raft_consensus.cc:348] T 54b3efabad7f4b608cc9d931546b5743 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } }
I20240501 07:59:08.518226 9654 raft_consensus.cc:374] T 54b3efabad7f4b608cc9d931546b5743 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:08.518268 9654 raft_consensus.cc:729] T 54b3efabad7f4b608cc9d931546b5743 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8315d22c56f7439a9e1fc82f153f018c, State: Initialized, Role: FOLLOWER
I20240501 07:59:08.518393 9654 consensus_queue.cc:260] T 54b3efabad7f4b608cc9d931546b5743 P 8315d22c56f7439a9e1fc82f153f018c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } }
I20240501 07:59:08.518672 9654 ts_tablet_manager.cc:1398] T 54b3efabad7f4b608cc9d931546b5743 P 8315d22c56f7439a9e1fc82f153f018c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:08.519596 8166 raft_consensus.cc:2932] T b7d57b7fd5234f50afbc859bc997f3e1 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { replace: true } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: true } } }
I20240501 07:59:08.519771 9651 tablet_bootstrap.cc:492] T 54b3efabad7f4b608cc9d931546b5743 P 1f9cb72be98941b5a8e504cbe34132b9: Bootstrap starting.
I20240501 07:59:08.520036 6707 raft_consensus.cc:2932] T b7d57b7fd5234f50afbc859bc997f3e1 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { replace: true } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: true } } }
I20240501 07:59:08.520942 9651 tablet_bootstrap.cc:654] T 54b3efabad7f4b608cc9d931546b5743 P 1f9cb72be98941b5a8e504cbe34132b9: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:08.521765 6314 raft_consensus.cc:2932] T b7d57b7fd5234f50afbc859bc997f3e1 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { replace: true } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: true } } }
I20240501 07:59:08.522495 6172 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55950:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 07:59:08.522887 6168 catalog_manager.cc:5484] T b7d57b7fd5234f50afbc859bc997f3e1 P 3b164cbbc00f4e8eb3254ddd349a3ff9 reported cstate change: config changed from index -1 to 4, NON_VOTER 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 07:59:08.528812 8924 raft_consensus.cc:1261] T 4cdabbbfd1df4a5a9683e3c8e569d624 P 13d05ebdca8c4f489c597ce550dd773d [term 1 FOLLOWER]: Refusing update from remote peer 83f20e84af4c40d09a4804b6b26e697f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:08.528899 8794 raft_consensus.cc:1261] T 4cdabbbfd1df4a5a9683e3c8e569d624 P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Refusing update from remote peer 83f20e84af4c40d09a4804b6b26e697f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:08.529392 9607 consensus_queue.cc:1035] T 4cdabbbfd1df4a5a9683e3c8e569d624 P 83f20e84af4c40d09a4804b6b26e697f [LEADER]: Connected to new peer: Peer: permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:08.529807 9607 consensus_queue.cc:1035] T 4cdabbbfd1df4a5a9683e3c8e569d624 P 83f20e84af4c40d09a4804b6b26e697f [LEADER]: Connected to new peer: Peer: permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:08.533228 6167 catalog_manager.cc:5576] TS 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134:40141) does not have the latest schema for tablet 4cdabbbfd1df4a5a9683e3c8e569d624 (table rebalancer_dynamic_table [id=5a2c4b25a3944383893b9fd726c21cad]). Expected version 1 got 0
I20240501 07:59:08.533708 6168 catalog_manager.cc:5576] TS 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133:39859) does not have the latest schema for tablet 4cdabbbfd1df4a5a9683e3c8e569d624 (table rebalancer_dynamic_table [id=5a2c4b25a3944383893b9fd726c21cad]). Expected version 1 got 0
I20240501 07:59:08.534097 9584 mvcc.cc:205] Tried to move back new op lower bound from 7022798227568545792 to 7022798227062280192. Current Snapshot: MvccSnapshot[applied={T|T < 7022798227568545792}]
I20240501 07:59:08.534431 9672 tablet.cc:1722] T 4cdabbbfd1df4a5a9683e3c8e569d624 P 83f20e84af4c40d09a4804b6b26e697f: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 07:59:08.534909 9672 tablet.cc:1620] T 4cdabbbfd1df4a5a9683e3c8e569d624 P 83f20e84af4c40d09a4804b6b26e697f: MemRowSet was empty: no flush needed.
I20240501 07:59:08.535354 6168 catalog_manager.cc:5576] TS 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149) does not have the latest schema for tablet 4cdabbbfd1df4a5a9683e3c8e569d624 (table rebalancer_dynamic_table [id=5a2c4b25a3944383893b9fd726c21cad]). Expected version 1 got 0
I20240501 07:59:08.537032 6166 catalog_manager.cc:5576] TS 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133:39859) does not have the latest schema for tablet 4cdabbbfd1df4a5a9683e3c8e569d624 (table rebalancer_dynamic_table [id=5a2c4b25a3944383893b9fd726c21cad]). Expected version 1 got 0
I20240501 07:59:08.537498 9580 mvcc.cc:205] Tried to move back new op lower bound from 7022798227568545792 to 7022798227062280192. Current Snapshot: MvccSnapshot[applied={T|T < 7022798227568545792}]
I20240501 07:59:08.537475 9676 tablet.cc:1722] T 4cdabbbfd1df4a5a9683e3c8e569d624 P 13d05ebdca8c4f489c597ce550dd773d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 07:59:08.538443 6166 catalog_manager.cc:5576] TS 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134:40141) does not have the latest schema for tablet 4cdabbbfd1df4a5a9683e3c8e569d624 (table rebalancer_dynamic_table [id=5a2c4b25a3944383893b9fd726c21cad]). Expected version 1 got 0
I20240501 07:59:08.538761 9676 tablet.cc:1620] T 4cdabbbfd1df4a5a9683e3c8e569d624 P 13d05ebdca8c4f489c597ce550dd773d: MemRowSet was empty: no flush needed.
I20240501 07:59:08.539882 9651 tablet_bootstrap.cc:492] T 54b3efabad7f4b608cc9d931546b5743 P 1f9cb72be98941b5a8e504cbe34132b9: No bootstrap required, opened a new log
I20240501 07:59:08.539907 6167 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 0bc706e85ce74cef97a4827ea4d4aa65: rebalancer_dynamic_table [id=5a2c4b25a3944383893b9fd726c21cad] alter complete (version 1)
I20240501 07:59:08.539986 9651 ts_tablet_manager.cc:1367] T 54b3efabad7f4b608cc9d931546b5743 P 1f9cb72be98941b5a8e504cbe34132b9: Time spent bootstrapping tablet: real 0.020s user 0.001s sys 0.002s
I20240501 07:59:08.539955 9675 tablet.cc:1722] T 4cdabbbfd1df4a5a9683e3c8e569d624 P 74077b61d6f54342bf9736b4a9f2eade: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 07:59:08.540470 9651 raft_consensus.cc:348] T 54b3efabad7f4b608cc9d931546b5743 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } }
I20240501 07:59:08.540581 9651 raft_consensus.cc:374] T 54b3efabad7f4b608cc9d931546b5743 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:08.540618 9651 raft_consensus.cc:729] T 54b3efabad7f4b608cc9d931546b5743 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f9cb72be98941b5a8e504cbe34132b9, State: Initialized, Role: FOLLOWER
I20240501 07:59:08.540751 9651 consensus_queue.cc:260] T 54b3efabad7f4b608cc9d931546b5743 P 1f9cb72be98941b5a8e504cbe34132b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } }
I20240501 07:59:08.540786 9675 tablet.cc:1620] T 4cdabbbfd1df4a5a9683e3c8e569d624 P 74077b61d6f54342bf9736b4a9f2eade: MemRowSet was empty: no flush needed.
I20240501 07:59:08.541054 9651 ts_tablet_manager.cc:1398] T 54b3efabad7f4b608cc9d931546b5743 P 1f9cb72be98941b5a8e504cbe34132b9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:08.552268 9116 raft_consensus.cc:482] T cfcc814f4d2d488ba745ecaf0cc6d61a P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:08.552392 9116 raft_consensus.cc:504] T cfcc814f4d2d488ba745ecaf0cc6d61a P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } }
I20240501 07:59:08.553038 9116 leader_election.cc:288] T cfcc814f4d2d488ba745ecaf0cc6d61a P 8315d22c56f7439a9e1fc82f153f018c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134:40141), a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607)
I20240501 07:59:08.553340 8924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfcc814f4d2d488ba745ecaf0cc6d61a" candidate_uuid: "8315d22c56f7439a9e1fc82f153f018c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13d05ebdca8c4f489c597ce550dd773d" is_pre_election: true
I20240501 07:59:08.553485 8924 raft_consensus.cc:2454] T cfcc814f4d2d488ba745ecaf0cc6d61a P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8315d22c56f7439a9e1fc82f153f018c in term 0.
I20240501 07:59:08.553512 9056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfcc814f4d2d488ba745ecaf0cc6d61a" candidate_uuid: "8315d22c56f7439a9e1fc82f153f018c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a505e7cb133c4392bfda8a4a20874ae9" is_pre_election: true
I20240501 07:59:08.553647 9056 raft_consensus.cc:2454] T cfcc814f4d2d488ba745ecaf0cc6d61a P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8315d22c56f7439a9e1fc82f153f018c in term 0.
I20240501 07:59:08.553990 6380 leader_election.cc:302] T cfcc814f4d2d488ba745ecaf0cc6d61a P 8315d22c56f7439a9e1fc82f153f018c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8315d22c56f7439a9e1fc82f153f018c, a505e7cb133c4392bfda8a4a20874ae9; no voters:
I20240501 07:59:08.554292 9116 raft_consensus.cc:2781] T cfcc814f4d2d488ba745ecaf0cc6d61a P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:08.554354 9116 raft_consensus.cc:482] T cfcc814f4d2d488ba745ecaf0cc6d61a P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:08.554399 9116 raft_consensus.cc:3037] T cfcc814f4d2d488ba745ecaf0cc6d61a P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:08.555550 9116 raft_consensus.cc:504] T cfcc814f4d2d488ba745ecaf0cc6d61a P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } }
I20240501 07:59:08.556093 9116 leader_election.cc:288] T cfcc814f4d2d488ba745ecaf0cc6d61a P 8315d22c56f7439a9e1fc82f153f018c [CANDIDATE]: Term 1 election: Requested vote from peers 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134:40141), a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607)
I20240501 07:59:08.556306 8924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfcc814f4d2d488ba745ecaf0cc6d61a" candidate_uuid: "8315d22c56f7439a9e1fc82f153f018c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13d05ebdca8c4f489c597ce550dd773d"
I20240501 07:59:08.556421 8924 raft_consensus.cc:3037] T cfcc814f4d2d488ba745ecaf0cc6d61a P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:08.556397 9056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfcc814f4d2d488ba745ecaf0cc6d61a" candidate_uuid: "8315d22c56f7439a9e1fc82f153f018c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a505e7cb133c4392bfda8a4a20874ae9"
I20240501 07:59:08.556594 9056 raft_consensus.cc:3037] T cfcc814f4d2d488ba745ecaf0cc6d61a P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:08.557476 8924 raft_consensus.cc:2454] T cfcc814f4d2d488ba745ecaf0cc6d61a P 13d05ebdca8c4f489c597ce550dd773d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8315d22c56f7439a9e1fc82f153f018c in term 1.
I20240501 07:59:08.557559 9056 raft_consensus.cc:2454] T cfcc814f4d2d488ba745ecaf0cc6d61a P a505e7cb133c4392bfda8a4a20874ae9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8315d22c56f7439a9e1fc82f153f018c in term 1.
I20240501 07:59:08.557842 6380 leader_election.cc:302] T cfcc814f4d2d488ba745ecaf0cc6d61a P 8315d22c56f7439a9e1fc82f153f018c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8315d22c56f7439a9e1fc82f153f018c, a505e7cb133c4392bfda8a4a20874ae9; no voters:
I20240501 07:59:08.558002 8512 raft_consensus.cc:2781] T cfcc814f4d2d488ba745ecaf0cc6d61a P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:08.558092 8512 raft_consensus.cc:686] T cfcc814f4d2d488ba745ecaf0cc6d61a P 8315d22c56f7439a9e1fc82f153f018c [term 1 LEADER]: Becoming Leader. State: Replica: 8315d22c56f7439a9e1fc82f153f018c, State: Running, Role: LEADER
I20240501 07:59:08.558223 8512 consensus_queue.cc:237] T cfcc814f4d2d488ba745ecaf0cc6d61a P 8315d22c56f7439a9e1fc82f153f018c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } }
I20240501 07:59:08.560549 6166 catalog_manager.cc:5484] T cfcc814f4d2d488ba745ecaf0cc6d61a P 8315d22c56f7439a9e1fc82f153f018c reported cstate change: term changed from 0 to 1, leader changed from <none> to 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130). New cstate: current_term: 1 leader_uuid: "8315d22c56f7439a9e1fc82f153f018c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } health_report { overall_health: UNKNOWN } } }
I20240501 07:59:08.577904 9116 raft_consensus.cc:482] T 54b3efabad7f4b608cc9d931546b5743 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:08.578042 9116 raft_consensus.cc:504] T 54b3efabad7f4b608cc9d931546b5743 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } }
I20240501 07:59:08.578634 9116 leader_election.cc:288] T 54b3efabad7f4b608cc9d931546b5743 P 8315d22c56f7439a9e1fc82f153f018c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132:36773), 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149)
I20240501 07:59:08.578984 6707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54b3efabad7f4b608cc9d931546b5743" candidate_uuid: "8315d22c56f7439a9e1fc82f153f018c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f9cb72be98941b5a8e504cbe34132b9" is_pre_election: true
I20240501 07:59:08.579133 6707 raft_consensus.cc:2454] T 54b3efabad7f4b608cc9d931546b5743 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8315d22c56f7439a9e1fc82f153f018c in term 0.
I20240501 07:59:08.579459 6381 leader_election.cc:302] T 54b3efabad7f4b608cc9d931546b5743 P 8315d22c56f7439a9e1fc82f153f018c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f9cb72be98941b5a8e504cbe34132b9, 8315d22c56f7439a9e1fc82f153f018c; no voters:
I20240501 07:59:08.579617 9116 raft_consensus.cc:2781] T 54b3efabad7f4b608cc9d931546b5743 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:08.579691 9116 raft_consensus.cc:482] T 54b3efabad7f4b608cc9d931546b5743 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:08.579739 9116 raft_consensus.cc:3037] T 54b3efabad7f4b608cc9d931546b5743 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:08.580826 9116 raft_consensus.cc:504] T 54b3efabad7f4b608cc9d931546b5743 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } }
I20240501 07:59:08.581194 9116 leader_election.cc:288] T 54b3efabad7f4b608cc9d931546b5743 P 8315d22c56f7439a9e1fc82f153f018c [CANDIDATE]: Term 1 election: Requested vote from peers 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132:36773), 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149)
I20240501 07:59:08.581775 6707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54b3efabad7f4b608cc9d931546b5743" candidate_uuid: "8315d22c56f7439a9e1fc82f153f018c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f9cb72be98941b5a8e504cbe34132b9"
I20240501 07:59:08.581894 6707 raft_consensus.cc:3037] T 54b3efabad7f4b608cc9d931546b5743 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:08.582701 6707 raft_consensus.cc:2454] T 54b3efabad7f4b608cc9d931546b5743 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8315d22c56f7439a9e1fc82f153f018c in term 1.
I20240501 07:59:08.582989 6381 leader_election.cc:302] T 54b3efabad7f4b608cc9d931546b5743 P 8315d22c56f7439a9e1fc82f153f018c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f9cb72be98941b5a8e504cbe34132b9, 8315d22c56f7439a9e1fc82f153f018c; no voters:
I20240501 07:59:08.583153 9116 raft_consensus.cc:2781] T 54b3efabad7f4b608cc9d931546b5743 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:08.583256 9116 raft_consensus.cc:686] T 54b3efabad7f4b608cc9d931546b5743 P 8315d22c56f7439a9e1fc82f153f018c [term 1 LEADER]: Becoming Leader. State: Replica: 8315d22c56f7439a9e1fc82f153f018c, State: Running, Role: LEADER
I20240501 07:59:08.583413 9116 consensus_queue.cc:237] T 54b3efabad7f4b608cc9d931546b5743 P 8315d22c56f7439a9e1fc82f153f018c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } }
I20240501 07:59:08.583832 9189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54b3efabad7f4b608cc9d931546b5743" candidate_uuid: "8315d22c56f7439a9e1fc82f153f018c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83f20e84af4c40d09a4804b6b26e697f" is_pre_election: true
I20240501 07:59:08.583958 9189 raft_consensus.cc:2454] T 54b3efabad7f4b608cc9d931546b5743 P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8315d22c56f7439a9e1fc82f153f018c in term 0.
I20240501 07:59:08.584194 9188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54b3efabad7f4b608cc9d931546b5743" candidate_uuid: "8315d22c56f7439a9e1fc82f153f018c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83f20e84af4c40d09a4804b6b26e697f"
I20240501 07:59:08.584307 9188 raft_consensus.cc:3037] T 54b3efabad7f4b608cc9d931546b5743 P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:08.585247 9188 raft_consensus.cc:2454] T 54b3efabad7f4b608cc9d931546b5743 P 83f20e84af4c40d09a4804b6b26e697f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8315d22c56f7439a9e1fc82f153f018c in term 1.
I20240501 07:59:08.587109 6166 catalog_manager.cc:5484] T 54b3efabad7f4b608cc9d931546b5743 P 8315d22c56f7439a9e1fc82f153f018c reported cstate change: term changed from 0 to 1, leader changed from <none> to 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130). New cstate: current_term: 1 leader_uuid: "8315d22c56f7439a9e1fc82f153f018c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } health_report { overall_health: UNKNOWN } } }
I20240501 07:59:08.589679 9673 raft_consensus.cc:482] T c5a876dfb598428ab350593129505504 P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:08.589795 9673 raft_consensus.cc:504] T c5a876dfb598428ab350593129505504 P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } }
I20240501 07:59:08.590452 9673 leader_election.cc:288] T c5a876dfb598428ab350593129505504 P 83f20e84af4c40d09a4804b6b26e697f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130:32941), 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133:39859)
I20240501 07:59:08.590986 8794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5a876dfb598428ab350593129505504" candidate_uuid: "83f20e84af4c40d09a4804b6b26e697f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74077b61d6f54342bf9736b4a9f2eade" is_pre_election: true
I20240501 07:59:08.591147 8794 raft_consensus.cc:2454] T c5a876dfb598428ab350593129505504 P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83f20e84af4c40d09a4804b6b26e697f in term 0.
I20240501 07:59:08.591511 9122 leader_election.cc:302] T c5a876dfb598428ab350593129505504 P 83f20e84af4c40d09a4804b6b26e697f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74077b61d6f54342bf9736b4a9f2eade, 83f20e84af4c40d09a4804b6b26e697f; no voters:
I20240501 07:59:08.591774 9673 raft_consensus.cc:2781] T c5a876dfb598428ab350593129505504 P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:08.591832 9673 raft_consensus.cc:482] T c5a876dfb598428ab350593129505504 P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:08.591869 9673 raft_consensus.cc:3037] T c5a876dfb598428ab350593129505504 P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:08.592885 9673 raft_consensus.cc:504] T c5a876dfb598428ab350593129505504 P 83f20e84af4c40d09a4804b6b26e697f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } }
I20240501 07:59:08.593273 9673 leader_election.cc:288] T c5a876dfb598428ab350593129505504 P 83f20e84af4c40d09a4804b6b26e697f [CANDIDATE]: Term 1 election: Requested vote from peers 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130:32941), 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133:39859)
I20240501 07:59:08.593708 8794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5a876dfb598428ab350593129505504" candidate_uuid: "83f20e84af4c40d09a4804b6b26e697f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74077b61d6f54342bf9736b4a9f2eade"
I20240501 07:59:08.593815 8794 raft_consensus.cc:3037] T c5a876dfb598428ab350593129505504 P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:08.594663 8794 raft_consensus.cc:2454] T c5a876dfb598428ab350593129505504 P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83f20e84af4c40d09a4804b6b26e697f in term 1.
I20240501 07:59:08.594667 6444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5a876dfb598428ab350593129505504" candidate_uuid: "83f20e84af4c40d09a4804b6b26e697f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8315d22c56f7439a9e1fc82f153f018c" is_pre_election: true
I20240501 07:59:08.594792 6444 raft_consensus.cc:2454] T c5a876dfb598428ab350593129505504 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83f20e84af4c40d09a4804b6b26e697f in term 0.
I20240501 07:59:08.595006 9122 leader_election.cc:302] T c5a876dfb598428ab350593129505504 P 83f20e84af4c40d09a4804b6b26e697f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74077b61d6f54342bf9736b4a9f2eade, 83f20e84af4c40d09a4804b6b26e697f; no voters:
I20240501 07:59:08.595041 6445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5a876dfb598428ab350593129505504" candidate_uuid: "83f20e84af4c40d09a4804b6b26e697f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8315d22c56f7439a9e1fc82f153f018c"
I20240501 07:59:08.595144 6445 raft_consensus.cc:3037] T c5a876dfb598428ab350593129505504 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:08.595171 9673 raft_consensus.cc:2781] T c5a876dfb598428ab350593129505504 P 83f20e84af4c40d09a4804b6b26e697f [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:08.595239 9673 raft_consensus.cc:686] T c5a876dfb598428ab350593129505504 P 83f20e84af4c40d09a4804b6b26e697f [term 1 LEADER]: Becoming Leader. State: Replica: 83f20e84af4c40d09a4804b6b26e697f, State: Running, Role: LEADER
I20240501 07:59:08.595379 9673 consensus_queue.cc:237] T c5a876dfb598428ab350593129505504 P 83f20e84af4c40d09a4804b6b26e697f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } }
I20240501 07:59:08.596182 6445 raft_consensus.cc:2454] T c5a876dfb598428ab350593129505504 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83f20e84af4c40d09a4804b6b26e697f in term 1.
I20240501 07:59:08.597766 6166 catalog_manager.cc:5484] T c5a876dfb598428ab350593129505504 P 83f20e84af4c40d09a4804b6b26e697f reported cstate change: term changed from 0 to 1, leader changed from <none> to 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136). New cstate: current_term: 1 leader_uuid: "83f20e84af4c40d09a4804b6b26e697f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } health_report { overall_health: UNKNOWN } } }
I20240501 07:59:08.632642 9740 ts_tablet_manager.cc:902] T d3f2250a62c6429e886753aaffcbe883 P 83f20e84af4c40d09a4804b6b26e697f: Initiating tablet copy from peer 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131:34471)
I20240501 07:59:08.633680 9740 tablet_copy_client.cc:308] T d3f2250a62c6429e886753aaffcbe883 P 83f20e84af4c40d09a4804b6b26e697f: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:34471
I20240501 07:59:08.638027 6597 tablet_copy_service.cc:140] P 3b164cbbc00f4e8eb3254ddd349a3ff9: Received BeginTabletCopySession request for tablet d3f2250a62c6429e886753aaffcbe883 from peer 83f20e84af4c40d09a4804b6b26e697f ({username='slave'} at 127.0.99.136:47269)
I20240501 07:59:08.638198 6597 tablet_copy_service.cc:161] P 3b164cbbc00f4e8eb3254ddd349a3ff9: Beginning new tablet copy session on tablet d3f2250a62c6429e886753aaffcbe883 from peer 83f20e84af4c40d09a4804b6b26e697f at {username='slave'} at 127.0.99.136:47269: session id = 83f20e84af4c40d09a4804b6b26e697f-d3f2250a62c6429e886753aaffcbe883
I20240501 07:59:08.639920 6597 tablet_copy_source_session.cc:205] T d3f2250a62c6429e886753aaffcbe883 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:08.641851 9740 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3f2250a62c6429e886753aaffcbe883. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:08.643407 6167 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55950:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 07:59:08.644361 9740 tablet_copy_client.cc:783] T d3f2250a62c6429e886753aaffcbe883 P 83f20e84af4c40d09a4804b6b26e697f: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:08.644580 9740 tablet_copy_client.cc:647] T d3f2250a62c6429e886753aaffcbe883 P 83f20e84af4c40d09a4804b6b26e697f: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:08.646852 9740 tablet_copy_client.cc:523] T d3f2250a62c6429e886753aaffcbe883 P 83f20e84af4c40d09a4804b6b26e697f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:08.648582 9740 tablet_bootstrap.cc:492] T d3f2250a62c6429e886753aaffcbe883 P 83f20e84af4c40d09a4804b6b26e697f: Bootstrap starting.
I20240501 07:59:08.649480 9672 tablet.cc:1722] T 4cdabbbfd1df4a5a9683e3c8e569d624 P 83f20e84af4c40d09a4804b6b26e697f: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 07:59:08.649788 9672 tablet.cc:1620] T 4cdabbbfd1df4a5a9683e3c8e569d624 P 83f20e84af4c40d09a4804b6b26e697f: MemRowSet was empty: no flush needed.
I20240501 07:59:08.649863 9676 tablet.cc:1722] T 4cdabbbfd1df4a5a9683e3c8e569d624 P 13d05ebdca8c4f489c597ce550dd773d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 07:59:08.650174 9676 tablet.cc:1620] T 4cdabbbfd1df4a5a9683e3c8e569d624 P 13d05ebdca8c4f489c597ce550dd773d: MemRowSet was empty: no flush needed.
I20240501 07:59:08.650405 9675 tablet.cc:1722] T 4cdabbbfd1df4a5a9683e3c8e569d624 P 74077b61d6f54342bf9736b4a9f2eade: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 07:59:08.650709 9675 tablet.cc:1620] T 4cdabbbfd1df4a5a9683e3c8e569d624 P 74077b61d6f54342bf9736b4a9f2eade: MemRowSet was empty: no flush needed.
I20240501 07:59:08.658849 6167 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 0bc706e85ce74cef97a4827ea4d4aa65: rebalancer_dynamic_table [id=5a2c4b25a3944383893b9fd726c21cad] alter complete (version 2)
I20240501 07:59:08.699781 9778 ts_tablet_manager.cc:902] T 271e3c5cf9934eaead7fa1ba942084be P 74077b61d6f54342bf9736b4a9f2eade: Initiating tablet copy from peer 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130:32941)
I20240501 07:59:08.700582 9778 tablet_copy_client.cc:308] T 271e3c5cf9934eaead7fa1ba942084be P 74077b61d6f54342bf9736b4a9f2eade: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:32941
I20240501 07:59:08.702126 9784 ts_tablet_manager.cc:902] T 810a2791eb04415dae47e85db87352cb P a505e7cb133c4392bfda8a4a20874ae9: Initiating tablet copy from peer 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132:36773)
I20240501 07:59:08.702883 9784 tablet_copy_client.cc:308] T 810a2791eb04415dae47e85db87352cb P a505e7cb133c4392bfda8a4a20874ae9: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:36773
I20240501 07:59:08.703287 9740 tablet_bootstrap.cc:492] T d3f2250a62c6429e886753aaffcbe883 P 83f20e84af4c40d09a4804b6b26e697f: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:08.703724 9740 tablet_bootstrap.cc:492] T d3f2250a62c6429e886753aaffcbe883 P 83f20e84af4c40d09a4804b6b26e697f: Bootstrap complete.
I20240501 07:59:08.703905 9740 ts_tablet_manager.cc:1367] T d3f2250a62c6429e886753aaffcbe883 P 83f20e84af4c40d09a4804b6b26e697f: Time spent bootstrapping tablet: real 0.055s user 0.030s sys 0.000s
I20240501 07:59:08.704406 9740 raft_consensus.cc:348] T d3f2250a62c6429e886753aaffcbe883 P 83f20e84af4c40d09a4804b6b26e697f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: true } }
I20240501 07:59:08.704552 9740 raft_consensus.cc:729] T d3f2250a62c6429e886753aaffcbe883 P 83f20e84af4c40d09a4804b6b26e697f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 83f20e84af4c40d09a4804b6b26e697f, State: Initialized, Role: LEARNER
I20240501 07:59:08.704684 9740 consensus_queue.cc:260] T d3f2250a62c6429e886753aaffcbe883 P 83f20e84af4c40d09a4804b6b26e697f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: true } }
I20240501 07:59:08.704854 9790 ts_tablet_manager.cc:902] T 26713549654249b09f820a31ee19e2b7 P a505e7cb133c4392bfda8a4a20874ae9: Initiating tablet copy from peer 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132:36773)
I20240501 07:59:08.705430 9790 tablet_copy_client.cc:308] T 26713549654249b09f820a31ee19e2b7 P a505e7cb133c4392bfda8a4a20874ae9: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:36773
I20240501 07:59:08.705667 9740 ts_tablet_manager.cc:1398] T d3f2250a62c6429e886753aaffcbe883 P 83f20e84af4c40d09a4804b6b26e697f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:59:08.706313 6597 tablet_copy_service.cc:342] P 3b164cbbc00f4e8eb3254ddd349a3ff9: Request end of tablet copy session 83f20e84af4c40d09a4804b6b26e697f-d3f2250a62c6429e886753aaffcbe883 received from {username='slave'} at 127.0.99.136:47269
I20240501 07:59:08.706465 6597 tablet_copy_service.cc:433] P 3b164cbbc00f4e8eb3254ddd349a3ff9: ending tablet copy session 83f20e84af4c40d09a4804b6b26e697f-d3f2250a62c6429e886753aaffcbe883 on tablet d3f2250a62c6429e886753aaffcbe883 with peer 83f20e84af4c40d09a4804b6b26e697f
I20240501 07:59:08.709702 6466 tablet_copy_service.cc:140] P 8315d22c56f7439a9e1fc82f153f018c: Received BeginTabletCopySession request for tablet 271e3c5cf9934eaead7fa1ba942084be from peer 74077b61d6f54342bf9736b4a9f2eade ({username='slave'} at 127.0.99.133:51179)
I20240501 07:59:08.709830 6466 tablet_copy_service.cc:161] P 8315d22c56f7439a9e1fc82f153f018c: Beginning new tablet copy session on tablet 271e3c5cf9934eaead7fa1ba942084be from peer 74077b61d6f54342bf9736b4a9f2eade at {username='slave'} at 127.0.99.133:51179: session id = 74077b61d6f54342bf9736b4a9f2eade-271e3c5cf9934eaead7fa1ba942084be
I20240501 07:59:08.710964 6466 tablet_copy_source_session.cc:205] T 271e3c5cf9934eaead7fa1ba942084be P 8315d22c56f7439a9e1fc82f153f018c: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:08.712550 9778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 271e3c5cf9934eaead7fa1ba942084be. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:08.715178 9778 tablet_copy_client.cc:783] T 271e3c5cf9934eaead7fa1ba942084be P 74077b61d6f54342bf9736b4a9f2eade: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:08.715549 9778 tablet_copy_client.cc:647] T 271e3c5cf9934eaead7fa1ba942084be P 74077b61d6f54342bf9736b4a9f2eade: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:08.716876 6727 tablet_copy_service.cc:140] P 1f9cb72be98941b5a8e504cbe34132b9: Received BeginTabletCopySession request for tablet 26713549654249b09f820a31ee19e2b7 from peer a505e7cb133c4392bfda8a4a20874ae9 ({username='slave'} at 127.0.99.135:56663)
I20240501 07:59:08.716876 6728 tablet_copy_service.cc:140] P 1f9cb72be98941b5a8e504cbe34132b9: Received BeginTabletCopySession request for tablet 810a2791eb04415dae47e85db87352cb from peer a505e7cb133c4392bfda8a4a20874ae9 ({username='slave'} at 127.0.99.135:56663)
I20240501 07:59:08.717061 6727 tablet_copy_service.cc:161] P 1f9cb72be98941b5a8e504cbe34132b9: Beginning new tablet copy session on tablet 26713549654249b09f820a31ee19e2b7 from peer a505e7cb133c4392bfda8a4a20874ae9 at {username='slave'} at 127.0.99.135:56663: session id = a505e7cb133c4392bfda8a4a20874ae9-26713549654249b09f820a31ee19e2b7
I20240501 07:59:08.717334 6728 tablet_copy_service.cc:161] P 1f9cb72be98941b5a8e504cbe34132b9: Beginning new tablet copy session on tablet 810a2791eb04415dae47e85db87352cb from peer a505e7cb133c4392bfda8a4a20874ae9 at {username='slave'} at 127.0.99.135:56663: session id = a505e7cb133c4392bfda8a4a20874ae9-810a2791eb04415dae47e85db87352cb
I20240501 07:59:08.718097 9778 tablet_copy_client.cc:523] T 271e3c5cf9934eaead7fa1ba942084be P 74077b61d6f54342bf9736b4a9f2eade: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:08.718506 6728 tablet_copy_source_session.cc:205] T 810a2791eb04415dae47e85db87352cb P 1f9cb72be98941b5a8e504cbe34132b9: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:08.719878 9778 tablet_bootstrap.cc:492] T 271e3c5cf9934eaead7fa1ba942084be P 74077b61d6f54342bf9736b4a9f2eade: Bootstrap starting.
I20240501 07:59:08.718506 6727 tablet_copy_source_session.cc:205] T 26713549654249b09f820a31ee19e2b7 P 1f9cb72be98941b5a8e504cbe34132b9: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:08.720150 9784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 810a2791eb04415dae47e85db87352cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:08.722419 9795 ts_tablet_manager.cc:902] T 6dc929e2a6ae45bd8b1f297583636bb6 P 74077b61d6f54342bf9736b4a9f2eade: Initiating tablet copy from peer 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130:32941)
I20240501 07:59:08.722613 9784 tablet_copy_client.cc:783] T 810a2791eb04415dae47e85db87352cb P a505e7cb133c4392bfda8a4a20874ae9: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:08.722822 9784 tablet_copy_client.cc:647] T 810a2791eb04415dae47e85db87352cb P a505e7cb133c4392bfda8a4a20874ae9: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:08.722997 9795 tablet_copy_client.cc:308] T 6dc929e2a6ae45bd8b1f297583636bb6 P 74077b61d6f54342bf9736b4a9f2eade: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:32941
I20240501 07:59:08.723460 6466 tablet_copy_service.cc:140] P 8315d22c56f7439a9e1fc82f153f018c: Received BeginTabletCopySession request for tablet 6dc929e2a6ae45bd8b1f297583636bb6 from peer 74077b61d6f54342bf9736b4a9f2eade ({username='slave'} at 127.0.99.133:51179)
I20240501 07:59:08.723568 6466 tablet_copy_service.cc:161] P 8315d22c56f7439a9e1fc82f153f018c: Beginning new tablet copy session on tablet 6dc929e2a6ae45bd8b1f297583636bb6 from peer 74077b61d6f54342bf9736b4a9f2eade at {username='slave'} at 127.0.99.133:51179: session id = 74077b61d6f54342bf9736b4a9f2eade-6dc929e2a6ae45bd8b1f297583636bb6
I20240501 07:59:08.724472 6466 tablet_copy_source_session.cc:205] T 6dc929e2a6ae45bd8b1f297583636bb6 P 8315d22c56f7439a9e1fc82f153f018c: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:08.724902 9790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26713549654249b09f820a31ee19e2b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:08.725298 9795 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6dc929e2a6ae45bd8b1f297583636bb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:08.726207 9784 tablet_copy_client.cc:523] T 810a2791eb04415dae47e85db87352cb P a505e7cb133c4392bfda8a4a20874ae9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:08.726509 9805 ts_tablet_manager.cc:902] T ec2d3ccf0937430f947f3e2bde0506c5 P 13d05ebdca8c4f489c597ce550dd773d: Initiating tablet copy from peer 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129:46875)
I20240501 07:59:08.727233 9795 tablet_copy_client.cc:783] T 6dc929e2a6ae45bd8b1f297583636bb6 P 74077b61d6f54342bf9736b4a9f2eade: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:08.727277 9790 tablet_copy_client.cc:783] T 26713549654249b09f820a31ee19e2b7 P a505e7cb133c4392bfda8a4a20874ae9: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:08.727466 9790 tablet_copy_client.cc:647] T 26713549654249b09f820a31ee19e2b7 P a505e7cb133c4392bfda8a4a20874ae9: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:08.727485 9795 tablet_copy_client.cc:647] T 6dc929e2a6ae45bd8b1f297583636bb6 P 74077b61d6f54342bf9736b4a9f2eade: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:08.728184 9784 tablet_bootstrap.cc:492] T 810a2791eb04415dae47e85db87352cb P a505e7cb133c4392bfda8a4a20874ae9: Bootstrap starting.
I20240501 07:59:08.728868 9790 tablet_copy_client.cc:523] T 26713549654249b09f820a31ee19e2b7 P a505e7cb133c4392bfda8a4a20874ae9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:08.729033 9805 tablet_copy_client.cc:308] T ec2d3ccf0937430f947f3e2bde0506c5 P 13d05ebdca8c4f489c597ce550dd773d: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:46875
I20240501 07:59:08.731170 9790 tablet_bootstrap.cc:492] T 26713549654249b09f820a31ee19e2b7 P a505e7cb133c4392bfda8a4a20874ae9: Bootstrap starting.
I20240501 07:59:08.739905 9795 tablet_copy_client.cc:523] T 6dc929e2a6ae45bd8b1f297583636bb6 P 74077b61d6f54342bf9736b4a9f2eade: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:08.741564 9795 tablet_bootstrap.cc:492] T 6dc929e2a6ae45bd8b1f297583636bb6 P 74077b61d6f54342bf9736b4a9f2eade: Bootstrap starting.
I20240501 07:59:08.760725 6172 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55950:
table { table_name: "rebalance_test_table_4" } modify_external_catalogs: true
I20240501 07:59:08.760907 6172 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55950:
table { table_name: "rebalance_test_table_4" } modify_external_catalogs: true
I20240501 07:59:08.763909 6166 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55950:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 07:59:08.765033 9778 tablet_bootstrap.cc:492] T 271e3c5cf9934eaead7fa1ba942084be P 74077b61d6f54342bf9736b4a9f2eade: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:08.765522 9778 tablet_bootstrap.cc:492] T 271e3c5cf9934eaead7fa1ba942084be P 74077b61d6f54342bf9736b4a9f2eade: Bootstrap complete.
I20240501 07:59:08.765832 9778 ts_tablet_manager.cc:1367] T 271e3c5cf9934eaead7fa1ba942084be P 74077b61d6f54342bf9736b4a9f2eade: Time spent bootstrapping tablet: real 0.046s user 0.033s sys 0.000s
I20240501 07:59:08.766369 9778 raft_consensus.cc:348] T 271e3c5cf9934eaead7fa1ba942084be P 74077b61d6f54342bf9736b4a9f2eade [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: true } }
I20240501 07:59:08.766580 9778 raft_consensus.cc:729] T 271e3c5cf9934eaead7fa1ba942084be P 74077b61d6f54342bf9736b4a9f2eade [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 74077b61d6f54342bf9736b4a9f2eade, State: Initialized, Role: LEARNER
I20240501 07:59:08.766757 9778 consensus_queue.cc:260] T 271e3c5cf9934eaead7fa1ba942084be P 74077b61d6f54342bf9736b4a9f2eade [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: true } }
I20240501 07:59:08.767474 6172 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 0bc706e85ce74cef97a4827ea4d4aa65: Sending DeleteTablet for 3 replicas of tablet 72b5270ae4704620be0a413139f41d26
I20240501 07:59:08.767758 9778 ts_tablet_manager.cc:1398] T 271e3c5cf9934eaead7fa1ba942084be P 74077b61d6f54342bf9736b4a9f2eade: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:59:08.767784 6172 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 0bc706e85ce74cef97a4827ea4d4aa65: Sending DeleteTablet for 3 replicas of tablet 5f0df3de857c49e48a52baa71163d161
I20240501 07:59:08.767997 6172 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 0bc706e85ce74cef97a4827ea4d4aa65: Sending DeleteTablet for 4 replicas of tablet 810a2791eb04415dae47e85db87352cb
I20240501 07:59:08.768033 6295 tablet_service.cc:1514] Processing DeleteTablet for tablet 72b5270ae4704620be0a413139f41d26 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:08 UTC) from {username='slave'} at 127.0.0.1:45480
I20240501 07:59:08.768337 9388 tablet_replica.cc:330] T 72b5270ae4704620be0a413139f41d26 P 8b8b3ad3aa8f4053b2167dd550f2086f: stopping tablet replica
I20240501 07:59:08.768494 9388 raft_consensus.cc:2229] T 72b5270ae4704620be0a413139f41d26 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:08.768503 6466 tablet_copy_service.cc:342] P 8315d22c56f7439a9e1fc82f153f018c: Request end of tablet copy session 74077b61d6f54342bf9736b4a9f2eade-271e3c5cf9934eaead7fa1ba942084be received from {username='slave'} at 127.0.99.133:51179
I20240501 07:59:08.768601 6466 tablet_copy_service.cc:433] P 8315d22c56f7439a9e1fc82f153f018c: ending tablet copy session 74077b61d6f54342bf9736b4a9f2eade-271e3c5cf9934eaead7fa1ba942084be on tablet 271e3c5cf9934eaead7fa1ba942084be with peer 74077b61d6f54342bf9736b4a9f2eade
I20240501 07:59:08.768640 9388 raft_consensus.cc:2258] T 72b5270ae4704620be0a413139f41d26 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:08.769136 6425 tablet_service.cc:1514] Processing DeleteTablet for tablet 72b5270ae4704620be0a413139f41d26 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:08 UTC) from {username='slave'} at 127.0.0.1:45034
I20240501 07:59:08.769222 9388 ts_tablet_manager.cc:1875] T 72b5270ae4704620be0a413139f41d26 P 8b8b3ad3aa8f4053b2167dd550f2086f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:59:08.769289 6424 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f0df3de857c49e48a52baa71163d161 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:08 UTC) from {username='slave'} at 127.0.0.1:45034
I20240501 07:59:08.769358 9386 tablet_replica.cc:330] T 72b5270ae4704620be0a413139f41d26 P 8315d22c56f7439a9e1fc82f153f018c: stopping tablet replica
I20240501 07:59:08.769445 6687 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f0df3de857c49e48a52baa71163d161 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:08 UTC) from {username='slave'} at 127.0.0.1:38524
I20240501 07:59:08.769487 9386 raft_consensus.cc:2229] T 72b5270ae4704620be0a413139f41d26 P 8315d22c56f7439a9e1fc82f153f018c [term 1 LEADER]: Raft consensus shutting down.
I20240501 07:59:08.769623 6686 tablet_service.cc:1514] Processing DeleteTablet for tablet 810a2791eb04415dae47e85db87352cb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:08 UTC) from {username='slave'} at 127.0.0.1:38524
I20240501 07:59:08.769693 9386 raft_consensus.cc:2258] T 72b5270ae4704620be0a413139f41d26 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:08.769767 9387 tablet_replica.cc:330] T 5f0df3de857c49e48a52baa71163d161 P 1f9cb72be98941b5a8e504cbe34132b9: stopping tablet replica
I20240501 07:59:08.769884 9387 raft_consensus.cc:2229] T 5f0df3de857c49e48a52baa71163d161 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:08.770009 9387 raft_consensus.cc:2258] T 5f0df3de857c49e48a52baa71163d161 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:08.770174 9386 ts_tablet_manager.cc:1875] T 72b5270ae4704620be0a413139f41d26 P 8315d22c56f7439a9e1fc82f153f018c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:59:08.770536 9387 ts_tablet_manager.cc:1875] T 5f0df3de857c49e48a52baa71163d161 P 1f9cb72be98941b5a8e504cbe34132b9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:59:08.771517 9388 ts_tablet_manager.cc:1888] T 72b5270ae4704620be0a413139f41d26 P 8b8b3ad3aa8f4053b2167dd550f2086f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 07:59:08.771587 9388 log.cc:1196] T 72b5270ae4704620be0a413139f41d26 P 8b8b3ad3aa8f4053b2167dd550f2086f: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/72b5270ae4704620be0a413139f41d26
I20240501 07:59:08.771852 9388 ts_tablet_manager.cc:1909] T 72b5270ae4704620be0a413139f41d26 P 8b8b3ad3aa8f4053b2167dd550f2086f: Deleting consensus metadata
I20240501 07:59:08.772174 9386 ts_tablet_manager.cc:1888] T 72b5270ae4704620be0a413139f41d26 P 8315d22c56f7439a9e1fc82f153f018c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 07:59:08.772241 9386 log.cc:1196] T 72b5270ae4704620be0a413139f41d26 P 8315d22c56f7439a9e1fc82f153f018c: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/72b5270ae4704620be0a413139f41d26
I20240501 07:59:08.772500 9386 ts_tablet_manager.cc:1909] T 72b5270ae4704620be0a413139f41d26 P 8315d22c56f7439a9e1fc82f153f018c: Deleting consensus metadata
I20240501 07:59:08.772715 9387 ts_tablet_manager.cc:1888] T 5f0df3de857c49e48a52baa71163d161 P 1f9cb72be98941b5a8e504cbe34132b9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 07:59:08.772778 9387 log.cc:1196] T 5f0df3de857c49e48a52baa71163d161 P 1f9cb72be98941b5a8e504cbe34132b9: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/5f0df3de857c49e48a52baa71163d161
I20240501 07:59:08.773255 9387 ts_tablet_manager.cc:1909] T 5f0df3de857c49e48a52baa71163d161 P 1f9cb72be98941b5a8e504cbe34132b9: Deleting consensus metadata
I20240501 07:59:08.773671 6555 tablet_service.cc:1514] Processing DeleteTablet for tablet 810a2791eb04415dae47e85db87352cb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:08 UTC) from {username='slave'} at 127.0.0.1:53874
I20240501 07:59:08.774092 9036 tablet_service.cc:1514] Processing DeleteTablet for tablet 810a2791eb04415dae47e85db87352cb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:08 UTC) from {username='slave'} at 127.0.0.1:44236
I20240501 07:59:08.774353 9387 tablet_replica.cc:330] T 810a2791eb04415dae47e85db87352cb P 1f9cb72be98941b5a8e504cbe34132b9: stopping tablet replica
I20240501 07:59:08.774508 9387 raft_consensus.cc:2229] T 810a2791eb04415dae47e85db87352cb P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 LEADER]: Raft consensus shutting down.
I20240501 07:59:08.773696 6556 tablet_service.cc:1514] Processing DeleteTablet for tablet 72b5270ae4704620be0a413139f41d26 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:08 UTC) from {username='slave'} at 127.0.0.1:53874
I20240501 07:59:08.774556 6159 catalog_manager.cc:4830] TS 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132:36773): tablet 5f0df3de857c49e48a52baa71163d161 (table rebalance_test_table_4 [id=e3d01900e54c4eb1920ccd2ff7a4f5db]) successfully deleted
I20240501 07:59:08.774791 9387 raft_consensus.cc:2258] T 810a2791eb04415dae47e85db87352cb P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:08.776312 6159 catalog_manager.cc:4830] TS 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130:32941): tablet 72b5270ae4704620be0a413139f41d26 (table rebalance_test_table_4 [id=e3d01900e54c4eb1920ccd2ff7a4f5db]) successfully deleted
I20240501 07:59:08.776619 6295 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f0df3de857c49e48a52baa71163d161 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:08 UTC) from {username='slave'} at 127.0.0.1:45480
I20240501 07:59:08.776669 9386 tablet_replica.cc:330] T 5f0df3de857c49e48a52baa71163d161 P 8315d22c56f7439a9e1fc82f153f018c: stopping tablet replica
I20240501 07:59:08.776808 9386 raft_consensus.cc:2229] T 5f0df3de857c49e48a52baa71163d161 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:08.776804 6294 tablet_service.cc:1514] Processing DeleteTablet for tablet 810a2791eb04415dae47e85db87352cb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:08 UTC) from {username='slave'} at 127.0.0.1:45480
I20240501 07:59:08.776947 9386 raft_consensus.cc:2258] T 5f0df3de857c49e48a52baa71163d161 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:08.777304 9740 ts_tablet_manager.cc:902] T 404bb81b9c404dd288d2f8160a4273da P 83f20e84af4c40d09a4804b6b26e697f: Initiating tablet copy from peer 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132:36773)
I20240501 07:59:08.777473 9386 ts_tablet_manager.cc:1875] T 5f0df3de857c49e48a52baa71163d161 P 8315d22c56f7439a9e1fc82f153f018c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:59:08.777757 9740 tablet_copy_client.cc:308] T 404bb81b9c404dd288d2f8160a4273da P 83f20e84af4c40d09a4804b6b26e697f: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:36773
I20240501 07:59:08.777959 9672 tablet.cc:1722] T 4cdabbbfd1df4a5a9683e3c8e569d624 P 83f20e84af4c40d09a4804b6b26e697f: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 07:59:08.778551 9675 tablet.cc:1722] T 4cdabbbfd1df4a5a9683e3c8e569d624 P 74077b61d6f54342bf9736b4a9f2eade: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 07:59:08.779783 9386 ts_tablet_manager.cc:1888] T 5f0df3de857c49e48a52baa71163d161 P 8315d22c56f7439a9e1fc82f153f018c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 07:59:08.779855 9386 log.cc:1196] T 5f0df3de857c49e48a52baa71163d161 P 8315d22c56f7439a9e1fc82f153f018c: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/5f0df3de857c49e48a52baa71163d161
I20240501 07:59:08.780108 9386 ts_tablet_manager.cc:1909] T 5f0df3de857c49e48a52baa71163d161 P 8315d22c56f7439a9e1fc82f153f018c: Deleting consensus metadata
I20240501 07:59:08.780882 6159 catalog_manager.cc:4830] TS 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130:32941): tablet 5f0df3de857c49e48a52baa71163d161 (table rebalance_test_table_4 [id=e3d01900e54c4eb1920ccd2ff7a4f5db]) successfully deleted
I20240501 07:59:08.781491 6157 catalog_manager.cc:4830] TS 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129:46875): tablet 72b5270ae4704620be0a413139f41d26 (table rebalance_test_table_4 [id=e3d01900e54c4eb1920ccd2ff7a4f5db]) successfully deleted
I20240501 07:59:08.781901 9388 tablet_replica.cc:330] T 5f0df3de857c49e48a52baa71163d161 P 8b8b3ad3aa8f4053b2167dd550f2086f: stopping tablet replica
I20240501 07:59:08.782083 9388 raft_consensus.cc:2229] T 5f0df3de857c49e48a52baa71163d161 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 LEADER]: Raft consensus shutting down.
I20240501 07:59:08.782486 9388 raft_consensus.cc:2258] T 5f0df3de857c49e48a52baa71163d161 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:08.783043 9388 ts_tablet_manager.cc:1875] T 5f0df3de857c49e48a52baa71163d161 P 8b8b3ad3aa8f4053b2167dd550f2086f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:59:08.785297 9676 tablet.cc:1722] T 4cdabbbfd1df4a5a9683e3c8e569d624 P 13d05ebdca8c4f489c597ce550dd773d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 07:59:08.785435 9388 ts_tablet_manager.cc:1888] T 5f0df3de857c49e48a52baa71163d161 P 8b8b3ad3aa8f4053b2167dd550f2086f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 07:59:08.785449 6166 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 0bc706e85ce74cef97a4827ea4d4aa65: rebalancer_dynamic_table_new_name [id=5a2c4b25a3944383893b9fd726c21cad] alter complete (version 3)
I20240501 07:59:08.785508 9388 log.cc:1196] T 5f0df3de857c49e48a52baa71163d161 P 8b8b3ad3aa8f4053b2167dd550f2086f: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/5f0df3de857c49e48a52baa71163d161
W20240501 07:59:08.787467 6158 catalog_manager.cc:4810] TS a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): delete failed for tablet 810a2791eb04415dae47e85db87352cb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 810a2791eb04415dae47e85db87352cb already in progress: copying tablet
I20240501 07:59:08.787595 6727 tablet_copy_service.cc:140] P 1f9cb72be98941b5a8e504cbe34132b9: Received BeginTabletCopySession request for tablet 404bb81b9c404dd288d2f8160a4273da from peer 83f20e84af4c40d09a4804b6b26e697f ({username='slave'} at 127.0.99.136:55471)
I20240501 07:59:08.787688 6727 tablet_copy_service.cc:161] P 1f9cb72be98941b5a8e504cbe34132b9: Beginning new tablet copy session on tablet 404bb81b9c404dd288d2f8160a4273da from peer 83f20e84af4c40d09a4804b6b26e697f at {username='slave'} at 127.0.99.136:55471: session id = 83f20e84af4c40d09a4804b6b26e697f-404bb81b9c404dd288d2f8160a4273da
I20240501 07:59:08.788611 6727 tablet_copy_source_session.cc:205] T 404bb81b9c404dd288d2f8160a4273da P 1f9cb72be98941b5a8e504cbe34132b9: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:08.789399 9387 ts_tablet_manager.cc:1875] T 810a2791eb04415dae47e85db87352cb P 1f9cb72be98941b5a8e504cbe34132b9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:59:08.789490 9740 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 404bb81b9c404dd288d2f8160a4273da. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:08.790639 9840 tablet_replica.cc:330] T 810a2791eb04415dae47e85db87352cb P 3b164cbbc00f4e8eb3254ddd349a3ff9: stopping tablet replica
I20240501 07:59:08.790802 9840 raft_consensus.cc:2229] T 810a2791eb04415dae47e85db87352cb P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:08.790988 9840 raft_consensus.cc:2258] T 810a2791eb04415dae47e85db87352cb P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:08.791571 9740 tablet_copy_client.cc:783] T 404bb81b9c404dd288d2f8160a4273da P 83f20e84af4c40d09a4804b6b26e697f: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:08.791738 9740 tablet_copy_client.cc:647] T 404bb81b9c404dd288d2f8160a4273da P 83f20e84af4c40d09a4804b6b26e697f: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:08.791755 9784 tablet_bootstrap.cc:492] T 810a2791eb04415dae47e85db87352cb P a505e7cb133c4392bfda8a4a20874ae9: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:08.791869 9387 ts_tablet_manager.cc:1888] T 810a2791eb04415dae47e85db87352cb P 1f9cb72be98941b5a8e504cbe34132b9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:59:08.791935 9387 log.cc:1196] T 810a2791eb04415dae47e85db87352cb P 1f9cb72be98941b5a8e504cbe34132b9: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/810a2791eb04415dae47e85db87352cb
I20240501 07:59:08.792062 9784 tablet_bootstrap.cc:492] T 810a2791eb04415dae47e85db87352cb P a505e7cb133c4392bfda8a4a20874ae9: Bootstrap complete.
I20240501 07:59:08.792176 9387 ts_tablet_manager.cc:1909] T 810a2791eb04415dae47e85db87352cb P 1f9cb72be98941b5a8e504cbe34132b9: Deleting consensus metadata
I20240501 07:59:08.792654 9388 ts_tablet_manager.cc:1909] T 5f0df3de857c49e48a52baa71163d161 P 8b8b3ad3aa8f4053b2167dd550f2086f: Deleting consensus metadata
I20240501 07:59:08.792763 6159 catalog_manager.cc:4830] TS 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132:36773): tablet 810a2791eb04415dae47e85db87352cb (table rebalance_test_table_4 [id=e3d01900e54c4eb1920ccd2ff7a4f5db]) successfully deleted
I20240501 07:59:08.792902 9840 ts_tablet_manager.cc:1875] T 810a2791eb04415dae47e85db87352cb P 3b164cbbc00f4e8eb3254ddd349a3ff9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:59:08.793192 9784 ts_tablet_manager.cc:1367] T 810a2791eb04415dae47e85db87352cb P a505e7cb133c4392bfda8a4a20874ae9: Time spent bootstrapping tablet: real 0.065s user 0.029s sys 0.005s
I20240501 07:59:08.793889 9740 tablet_copy_client.cc:523] T 404bb81b9c404dd288d2f8160a4273da P 83f20e84af4c40d09a4804b6b26e697f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:08.793892 6157 catalog_manager.cc:4830] TS 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129:46875): tablet 5f0df3de857c49e48a52baa71163d161 (table rebalance_test_table_4 [id=e3d01900e54c4eb1920ccd2ff7a4f5db]) successfully deleted
I20240501 07:59:08.793809 9388 tablet_replica.cc:330] T 810a2791eb04415dae47e85db87352cb P 8b8b3ad3aa8f4053b2167dd550f2086f: stopping tablet replica
I20240501 07:59:08.794174 9388 raft_consensus.cc:2229] T 810a2791eb04415dae47e85db87352cb P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:08.794425 9388 raft_consensus.cc:2258] T 810a2791eb04415dae47e85db87352cb P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:08.795161 9740 tablet_bootstrap.cc:492] T 404bb81b9c404dd288d2f8160a4273da P 83f20e84af4c40d09a4804b6b26e697f: Bootstrap starting.
I20240501 07:59:08.795625 9388 ts_tablet_manager.cc:1875] T 810a2791eb04415dae47e85db87352cb P 8b8b3ad3aa8f4053b2167dd550f2086f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:59:08.795770 6335 tablet_copy_service.cc:140] P 8b8b3ad3aa8f4053b2167dd550f2086f: Received BeginTabletCopySession request for tablet ec2d3ccf0937430f947f3e2bde0506c5 from peer 13d05ebdca8c4f489c597ce550dd773d ({username='slave'} at 127.0.99.134:55617)
I20240501 07:59:08.795889 6335 tablet_copy_service.cc:161] P 8b8b3ad3aa8f4053b2167dd550f2086f: Beginning new tablet copy session on tablet ec2d3ccf0937430f947f3e2bde0506c5 from peer 13d05ebdca8c4f489c597ce550dd773d at {username='slave'} at 127.0.99.134:55617: session id = 13d05ebdca8c4f489c597ce550dd773d-ec2d3ccf0937430f947f3e2bde0506c5
I20240501 07:59:08.797307 6335 tablet_copy_source_session.cc:205] T ec2d3ccf0937430f947f3e2bde0506c5 P 8b8b3ad3aa8f4053b2167dd550f2086f: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:08.799094 9805 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec2d3ccf0937430f947f3e2bde0506c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:08.801991 9805 tablet_copy_client.cc:783] T ec2d3ccf0937430f947f3e2bde0506c5 P 13d05ebdca8c4f489c597ce550dd773d: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:08.802167 9805 tablet_copy_client.cc:647] T ec2d3ccf0937430f947f3e2bde0506c5 P 13d05ebdca8c4f489c597ce550dd773d: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:08.802899 9388 ts_tablet_manager.cc:1888] T 810a2791eb04415dae47e85db87352cb P 8b8b3ad3aa8f4053b2167dd550f2086f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:59:08.803296 9388 log.cc:1196] T 810a2791eb04415dae47e85db87352cb P 8b8b3ad3aa8f4053b2167dd550f2086f: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/810a2791eb04415dae47e85db87352cb
I20240501 07:59:08.803699 9388 ts_tablet_manager.cc:1909] T 810a2791eb04415dae47e85db87352cb P 8b8b3ad3aa8f4053b2167dd550f2086f: Deleting consensus metadata
I20240501 07:59:08.804405 6157 catalog_manager.cc:4830] TS 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129:46875): tablet 810a2791eb04415dae47e85db87352cb (table rebalance_test_table_4 [id=e3d01900e54c4eb1920ccd2ff7a4f5db]) successfully deleted
I20240501 07:59:08.804819 9805 tablet_copy_client.cc:523] T ec2d3ccf0937430f947f3e2bde0506c5 P 13d05ebdca8c4f489c597ce550dd773d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:08.806555 9805 tablet_bootstrap.cc:492] T ec2d3ccf0937430f947f3e2bde0506c5 P 13d05ebdca8c4f489c597ce550dd773d: Bootstrap starting.
I20240501 07:59:08.807976 9784 raft_consensus.cc:348] T 810a2791eb04415dae47e85db87352cb P a505e7cb133c4392bfda8a4a20874ae9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: true } }
I20240501 07:59:08.808136 9784 raft_consensus.cc:729] T 810a2791eb04415dae47e85db87352cb P a505e7cb133c4392bfda8a4a20874ae9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a505e7cb133c4392bfda8a4a20874ae9, State: Initialized, Role: LEARNER
I20240501 07:59:08.808251 9784 consensus_queue.cc:260] T 810a2791eb04415dae47e85db87352cb P a505e7cb133c4392bfda8a4a20874ae9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: true } }
I20240501 07:59:08.808553 9784 ts_tablet_manager.cc:1398] T 810a2791eb04415dae47e85db87352cb P a505e7cb133c4392bfda8a4a20874ae9: Time spent starting tablet: real 0.013s user 0.000s sys 0.000s
I20240501 07:59:08.809212 6727 tablet_copy_service.cc:342] P 1f9cb72be98941b5a8e504cbe34132b9: Request end of tablet copy session a505e7cb133c4392bfda8a4a20874ae9-810a2791eb04415dae47e85db87352cb received from {username='slave'} at 127.0.99.135:56663
I20240501 07:59:08.809319 6727 tablet_copy_service.cc:433] P 1f9cb72be98941b5a8e504cbe34132b9: ending tablet copy session a505e7cb133c4392bfda8a4a20874ae9-810a2791eb04415dae47e85db87352cb on tablet 810a2791eb04415dae47e85db87352cb with peer a505e7cb133c4392bfda8a4a20874ae9
I20240501 07:59:08.810555 9036 tablet_service.cc:1514] Processing DeleteTablet for tablet 810a2791eb04415dae47e85db87352cb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:08 UTC) from {username='slave'} at 127.0.0.1:44236
I20240501 07:59:08.810830 9841 tablet_replica.cc:330] T 810a2791eb04415dae47e85db87352cb P a505e7cb133c4392bfda8a4a20874ae9: stopping tablet replica
I20240501 07:59:08.810997 9841 raft_consensus.cc:2229] T 810a2791eb04415dae47e85db87352cb P a505e7cb133c4392bfda8a4a20874ae9 [term 1 LEARNER]: Raft consensus shutting down.
I20240501 07:59:08.811165 9841 raft_consensus.cc:2258] T 810a2791eb04415dae47e85db87352cb P a505e7cb133c4392bfda8a4a20874ae9 [term 1 LEARNER]: Raft consensus is shut down!
I20240501 07:59:08.811957 9841 ts_tablet_manager.cc:1875] T 810a2791eb04415dae47e85db87352cb P a505e7cb133c4392bfda8a4a20874ae9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:59:08.812896 9840 ts_tablet_manager.cc:1888] T 810a2791eb04415dae47e85db87352cb P 3b164cbbc00f4e8eb3254ddd349a3ff9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:59:08.813073 9840 log.cc:1196] T 810a2791eb04415dae47e85db87352cb P 3b164cbbc00f4e8eb3254ddd349a3ff9: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/810a2791eb04415dae47e85db87352cb
I20240501 07:59:08.813437 9840 ts_tablet_manager.cc:1909] T 810a2791eb04415dae47e85db87352cb P 3b164cbbc00f4e8eb3254ddd349a3ff9: Deleting consensus metadata
I20240501 07:59:08.814090 9840 tablet_replica.cc:330] T 72b5270ae4704620be0a413139f41d26 P 3b164cbbc00f4e8eb3254ddd349a3ff9: stopping tablet replica
I20240501 07:59:08.814222 6160 catalog_manager.cc:4830] TS 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131:34471): tablet 810a2791eb04415dae47e85db87352cb (table rebalance_test_table_4 [id=e3d01900e54c4eb1920ccd2ff7a4f5db]) successfully deleted
I20240501 07:59:08.814291 9840 raft_consensus.cc:2229] T 72b5270ae4704620be0a413139f41d26 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:08.814514 9840 raft_consensus.cc:2258] T 72b5270ae4704620be0a413139f41d26 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:08.815155 9840 ts_tablet_manager.cc:1875] T 72b5270ae4704620be0a413139f41d26 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:59:08.815411 9841 ts_tablet_manager.cc:1888] T 810a2791eb04415dae47e85db87352cb P a505e7cb133c4392bfda8a4a20874ae9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:59:08.815546 9841 log.cc:1196] T 810a2791eb04415dae47e85db87352cb P a505e7cb133c4392bfda8a4a20874ae9: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/810a2791eb04415dae47e85db87352cb
I20240501 07:59:08.815891 9841 ts_tablet_manager.cc:1909] T 810a2791eb04415dae47e85db87352cb P a505e7cb133c4392bfda8a4a20874ae9: Deleting consensus metadata
I20240501 07:59:08.816723 9846 ts_tablet_manager.cc:902] T 187940ba79f546b1ad79e7ec1de76442 P 13d05ebdca8c4f489c597ce550dd773d: Initiating tablet copy from peer 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132:36773)
I20240501 07:59:08.816888 6158 catalog_manager.cc:4830] TS a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): tablet 810a2791eb04415dae47e85db87352cb (table rebalance_test_table_4 [id=e3d01900e54c4eb1920ccd2ff7a4f5db]) successfully deleted
I20240501 07:59:08.817760 9840 ts_tablet_manager.cc:1888] T 72b5270ae4704620be0a413139f41d26 P 3b164cbbc00f4e8eb3254ddd349a3ff9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 07:59:08.818202 9846 tablet_copy_client.cc:308] T 187940ba79f546b1ad79e7ec1de76442 P 13d05ebdca8c4f489c597ce550dd773d: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:36773
I20240501 07:59:08.818398 9840 log.cc:1196] T 72b5270ae4704620be0a413139f41d26 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/72b5270ae4704620be0a413139f41d26
I20240501 07:59:08.818765 9840 ts_tablet_manager.cc:1909] T 72b5270ae4704620be0a413139f41d26 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Deleting consensus metadata
I20240501 07:59:08.819522 6160 catalog_manager.cc:4830] TS 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131:34471): tablet 72b5270ae4704620be0a413139f41d26 (table rebalance_test_table_4 [id=e3d01900e54c4eb1920ccd2ff7a4f5db]) successfully deleted
I20240501 07:59:08.827745 9795 tablet_bootstrap.cc:492] T 6dc929e2a6ae45bd8b1f297583636bb6 P 74077b61d6f54342bf9736b4a9f2eade: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:08.827885 9790 tablet_bootstrap.cc:492] T 26713549654249b09f820a31ee19e2b7 P a505e7cb133c4392bfda8a4a20874ae9: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:08.828131 9795 tablet_bootstrap.cc:492] T 6dc929e2a6ae45bd8b1f297583636bb6 P 74077b61d6f54342bf9736b4a9f2eade: Bootstrap complete.
I20240501 07:59:08.828259 9790 tablet_bootstrap.cc:492] T 26713549654249b09f820a31ee19e2b7 P a505e7cb133c4392bfda8a4a20874ae9: Bootstrap complete.
I20240501 07:59:08.828473 9795 ts_tablet_manager.cc:1367] T 6dc929e2a6ae45bd8b1f297583636bb6 P 74077b61d6f54342bf9736b4a9f2eade: Time spent bootstrapping tablet: real 0.087s user 0.023s sys 0.005s
I20240501 07:59:08.828570 9790 ts_tablet_manager.cc:1367] T 26713549654249b09f820a31ee19e2b7 P a505e7cb133c4392bfda8a4a20874ae9: Time spent bootstrapping tablet: real 0.097s user 0.029s sys 0.000s
I20240501 07:59:08.829034 9790 raft_consensus.cc:348] T 26713549654249b09f820a31ee19e2b7 P a505e7cb133c4392bfda8a4a20874ae9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { replace: true } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: true } }
I20240501 07:59:08.829032 9795 raft_consensus.cc:348] T 6dc929e2a6ae45bd8b1f297583636bb6 P 74077b61d6f54342bf9736b4a9f2eade [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: true } }
I20240501 07:59:08.829161 9795 raft_consensus.cc:729] T 6dc929e2a6ae45bd8b1f297583636bb6 P 74077b61d6f54342bf9736b4a9f2eade [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 74077b61d6f54342bf9736b4a9f2eade, State: Initialized, Role: LEARNER
I20240501 07:59:08.829255 9790 raft_consensus.cc:729] T 26713549654249b09f820a31ee19e2b7 P a505e7cb133c4392bfda8a4a20874ae9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a505e7cb133c4392bfda8a4a20874ae9, State: Initialized, Role: LEARNER
I20240501 07:59:08.829309 9795 consensus_queue.cc:260] T 6dc929e2a6ae45bd8b1f297583636bb6 P 74077b61d6f54342bf9736b4a9f2eade [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: true } }
I20240501 07:59:08.829409 9790 consensus_queue.cc:260] T 26713549654249b09f820a31ee19e2b7 P a505e7cb133c4392bfda8a4a20874ae9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { replace: true } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: true } }
I20240501 07:59:08.829649 9795 ts_tablet_manager.cc:1398] T 6dc929e2a6ae45bd8b1f297583636bb6 P 74077b61d6f54342bf9736b4a9f2eade: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:08.830310 9790 ts_tablet_manager.cc:1398] T 26713549654249b09f820a31ee19e2b7 P a505e7cb133c4392bfda8a4a20874ae9: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:59:08.830554 6466 tablet_copy_service.cc:342] P 8315d22c56f7439a9e1fc82f153f018c: Request end of tablet copy session 74077b61d6f54342bf9736b4a9f2eade-6dc929e2a6ae45bd8b1f297583636bb6 received from {username='slave'} at 127.0.99.133:51179
I20240501 07:59:08.830658 6466 tablet_copy_service.cc:433] P 8315d22c56f7439a9e1fc82f153f018c: ending tablet copy session 74077b61d6f54342bf9736b4a9f2eade-6dc929e2a6ae45bd8b1f297583636bb6 on tablet 6dc929e2a6ae45bd8b1f297583636bb6 with peer 74077b61d6f54342bf9736b4a9f2eade
I20240501 07:59:08.830859 6727 tablet_copy_service.cc:342] P 1f9cb72be98941b5a8e504cbe34132b9: Request end of tablet copy session a505e7cb133c4392bfda8a4a20874ae9-26713549654249b09f820a31ee19e2b7 received from {username='slave'} at 127.0.99.135:56663
I20240501 07:59:08.830946 6727 tablet_copy_service.cc:433] P 1f9cb72be98941b5a8e504cbe34132b9: ending tablet copy session a505e7cb133c4392bfda8a4a20874ae9-26713549654249b09f820a31ee19e2b7 on tablet 26713549654249b09f820a31ee19e2b7 with peer a505e7cb133c4392bfda8a4a20874ae9
I20240501 07:59:08.832924 9860 ts_tablet_manager.cc:902] T bab34f083c8045e4a6df373db5ebd54b P 13d05ebdca8c4f489c597ce550dd773d: Initiating tablet copy from peer 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130:32941)
I20240501 07:59:08.833557 9860 tablet_copy_client.cc:308] T bab34f083c8045e4a6df373db5ebd54b P 13d05ebdca8c4f489c597ce550dd773d: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:32941
I20240501 07:59:08.834764 9795 ts_tablet_manager.cc:902] T 5ee738571af740c1b494d4f17413d1d8 P 74077b61d6f54342bf9736b4a9f2eade: Initiating tablet copy from peer 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129:46875)
I20240501 07:59:08.834975 9790 ts_tablet_manager.cc:902] T 3b9b71d44f754353b2321e8c36dd4dfd P a505e7cb133c4392bfda8a4a20874ae9: Initiating tablet copy from peer 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132:36773)
I20240501 07:59:08.835255 9795 tablet_copy_client.cc:308] T 5ee738571af740c1b494d4f17413d1d8 P 74077b61d6f54342bf9736b4a9f2eade: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:46875
I20240501 07:59:08.835439 9790 tablet_copy_client.cc:308] T 3b9b71d44f754353b2321e8c36dd4dfd P a505e7cb133c4392bfda8a4a20874ae9: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:36773
I20240501 07:59:08.835870 6727 tablet_copy_service.cc:140] P 1f9cb72be98941b5a8e504cbe34132b9: Received BeginTabletCopySession request for tablet 3b9b71d44f754353b2321e8c36dd4dfd from peer a505e7cb133c4392bfda8a4a20874ae9 ({username='slave'} at 127.0.99.135:56663)
I20240501 07:59:08.835968 6727 tablet_copy_service.cc:161] P 1f9cb72be98941b5a8e504cbe34132b9: Beginning new tablet copy session on tablet 3b9b71d44f754353b2321e8c36dd4dfd from peer a505e7cb133c4392bfda8a4a20874ae9 at {username='slave'} at 127.0.99.135:56663: session id = a505e7cb133c4392bfda8a4a20874ae9-3b9b71d44f754353b2321e8c36dd4dfd
I20240501 07:59:08.836907 6727 tablet_copy_source_session.cc:205] T 3b9b71d44f754353b2321e8c36dd4dfd P 1f9cb72be98941b5a8e504cbe34132b9: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:08.837983 9790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b9b71d44f754353b2321e8c36dd4dfd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:08.840036 6335 tablet_copy_service.cc:140] P 8b8b3ad3aa8f4053b2167dd550f2086f: Received BeginTabletCopySession request for tablet 5ee738571af740c1b494d4f17413d1d8 from peer 74077b61d6f54342bf9736b4a9f2eade ({username='slave'} at 127.0.99.133:50111)
I20240501 07:59:08.840158 6335 tablet_copy_service.cc:161] P 8b8b3ad3aa8f4053b2167dd550f2086f: Beginning new tablet copy session on tablet 5ee738571af740c1b494d4f17413d1d8 from peer 74077b61d6f54342bf9736b4a9f2eade at {username='slave'} at 127.0.99.133:50111: session id = 74077b61d6f54342bf9736b4a9f2eade-5ee738571af740c1b494d4f17413d1d8
I20240501 07:59:08.841208 6335 tablet_copy_source_session.cc:205] T 5ee738571af740c1b494d4f17413d1d8 P 8b8b3ad3aa8f4053b2167dd550f2086f: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:08.842118 9795 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ee738571af740c1b494d4f17413d1d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:08.842613 9790 tablet_copy_client.cc:783] T 3b9b71d44f754353b2321e8c36dd4dfd P a505e7cb133c4392bfda8a4a20874ae9: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:08.842823 9790 tablet_copy_client.cc:647] T 3b9b71d44f754353b2321e8c36dd4dfd P a505e7cb133c4392bfda8a4a20874ae9: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:08.843904 9790 tablet_copy_client.cc:523] T 3b9b71d44f754353b2321e8c36dd4dfd P a505e7cb133c4392bfda8a4a20874ae9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:08.844367 9795 tablet_copy_client.cc:783] T 5ee738571af740c1b494d4f17413d1d8 P 74077b61d6f54342bf9736b4a9f2eade: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:08.844579 9795 tablet_copy_client.cc:647] T 5ee738571af740c1b494d4f17413d1d8 P 74077b61d6f54342bf9736b4a9f2eade: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:08.845216 9790 tablet_bootstrap.cc:492] T 3b9b71d44f754353b2321e8c36dd4dfd P a505e7cb133c4392bfda8a4a20874ae9: Bootstrap starting.
I20240501 07:59:08.849253 9854 ts_tablet_manager.cc:902] T 6e2ac8c8857d45b88d3b51147d2609aa P 83f20e84af4c40d09a4804b6b26e697f: Initiating tablet copy from peer 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131:34471)
I20240501 07:59:08.850497 9854 tablet_copy_client.cc:308] T 6e2ac8c8857d45b88d3b51147d2609aa P 83f20e84af4c40d09a4804b6b26e697f: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:34471
I20240501 07:59:08.851279 6597 tablet_copy_service.cc:140] P 3b164cbbc00f4e8eb3254ddd349a3ff9: Received BeginTabletCopySession request for tablet 6e2ac8c8857d45b88d3b51147d2609aa from peer 83f20e84af4c40d09a4804b6b26e697f ({username='slave'} at 127.0.99.136:47269)
I20240501 07:59:08.851400 6597 tablet_copy_service.cc:161] P 3b164cbbc00f4e8eb3254ddd349a3ff9: Beginning new tablet copy session on tablet 6e2ac8c8857d45b88d3b51147d2609aa from peer 83f20e84af4c40d09a4804b6b26e697f at {username='slave'} at 127.0.99.136:47269: session id = 83f20e84af4c40d09a4804b6b26e697f-6e2ac8c8857d45b88d3b51147d2609aa
I20240501 07:59:08.852555 6597 tablet_copy_source_session.cc:205] T 6e2ac8c8857d45b88d3b51147d2609aa P 3b164cbbc00f4e8eb3254ddd349a3ff9: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:08.853480 9854 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e2ac8c8857d45b88d3b51147d2609aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:08.856846 6727 tablet_copy_service.cc:140] P 1f9cb72be98941b5a8e504cbe34132b9: Received BeginTabletCopySession request for tablet 187940ba79f546b1ad79e7ec1de76442 from peer 13d05ebdca8c4f489c597ce550dd773d ({username='slave'} at 127.0.99.134:44489)
I20240501 07:59:08.856977 6727 tablet_copy_service.cc:161] P 1f9cb72be98941b5a8e504cbe34132b9: Beginning new tablet copy session on tablet 187940ba79f546b1ad79e7ec1de76442 from peer 13d05ebdca8c4f489c597ce550dd773d at {username='slave'} at 127.0.99.134:44489: session id = 13d05ebdca8c4f489c597ce550dd773d-187940ba79f546b1ad79e7ec1de76442
I20240501 07:59:08.858059 6727 tablet_copy_source_session.cc:205] T 187940ba79f546b1ad79e7ec1de76442 P 1f9cb72be98941b5a8e504cbe34132b9: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:08.858881 9854 tablet_copy_client.cc:783] T 6e2ac8c8857d45b88d3b51147d2609aa P 83f20e84af4c40d09a4804b6b26e697f: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:08.859001 9846 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 187940ba79f546b1ad79e7ec1de76442. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:08.859114 9854 tablet_copy_client.cc:647] T 6e2ac8c8857d45b88d3b51147d2609aa P 83f20e84af4c40d09a4804b6b26e697f: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:08.861294 6466 tablet_copy_service.cc:140] P 8315d22c56f7439a9e1fc82f153f018c: Received BeginTabletCopySession request for tablet bab34f083c8045e4a6df373db5ebd54b from peer 13d05ebdca8c4f489c597ce550dd773d ({username='slave'} at 127.0.99.134:41693)
I20240501 07:59:08.861421 6466 tablet_copy_service.cc:161] P 8315d22c56f7439a9e1fc82f153f018c: Beginning new tablet copy session on tablet bab34f083c8045e4a6df373db5ebd54b from peer 13d05ebdca8c4f489c597ce550dd773d at {username='slave'} at 127.0.99.134:41693: session id = 13d05ebdca8c4f489c597ce550dd773d-bab34f083c8045e4a6df373db5ebd54b
I20240501 07:59:08.861590 9846 tablet_copy_client.cc:783] T 187940ba79f546b1ad79e7ec1de76442 P 13d05ebdca8c4f489c597ce550dd773d: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:08.861796 9846 tablet_copy_client.cc:647] T 187940ba79f546b1ad79e7ec1de76442 P 13d05ebdca8c4f489c597ce550dd773d: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:08.862457 6466 tablet_copy_source_session.cc:205] T bab34f083c8045e4a6df373db5ebd54b P 8315d22c56f7439a9e1fc82f153f018c: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:08.863461 9860 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bab34f083c8045e4a6df373db5ebd54b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:08.865798 9846 tablet_copy_client.cc:523] T 187940ba79f546b1ad79e7ec1de76442 P 13d05ebdca8c4f489c597ce550dd773d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:08.865983 9860 tablet_copy_client.cc:783] T bab34f083c8045e4a6df373db5ebd54b P 13d05ebdca8c4f489c597ce550dd773d: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:08.866117 9860 tablet_copy_client.cc:647] T bab34f083c8045e4a6df373db5ebd54b P 13d05ebdca8c4f489c597ce550dd773d: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:08.867125 9846 tablet_bootstrap.cc:492] T 187940ba79f546b1ad79e7ec1de76442 P 13d05ebdca8c4f489c597ce550dd773d: Bootstrap starting.
I20240501 07:59:08.867156 9860 tablet_copy_client.cc:523] T bab34f083c8045e4a6df373db5ebd54b P 13d05ebdca8c4f489c597ce550dd773d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:08.868284 9860 tablet_bootstrap.cc:492] T bab34f083c8045e4a6df373db5ebd54b P 13d05ebdca8c4f489c597ce550dd773d: Bootstrap starting.
I20240501 07:59:08.870502 9805 tablet_bootstrap.cc:492] T ec2d3ccf0937430f947f3e2bde0506c5 P 13d05ebdca8c4f489c597ce550dd773d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:08.871070 9805 tablet_bootstrap.cc:492] T ec2d3ccf0937430f947f3e2bde0506c5 P 13d05ebdca8c4f489c597ce550dd773d: Bootstrap complete.
I20240501 07:59:08.871358 9805 ts_tablet_manager.cc:1367] T ec2d3ccf0937430f947f3e2bde0506c5 P 13d05ebdca8c4f489c597ce550dd773d: Time spent bootstrapping tablet: real 0.065s user 0.028s sys 0.004s
I20240501 07:59:08.871842 9805 raft_consensus.cc:348] T ec2d3ccf0937430f947f3e2bde0506c5 P 13d05ebdca8c4f489c597ce550dd773d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { replace: true } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: true } }
I20240501 07:59:08.872014 9805 raft_consensus.cc:729] T ec2d3ccf0937430f947f3e2bde0506c5 P 13d05ebdca8c4f489c597ce550dd773d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 13d05ebdca8c4f489c597ce550dd773d, State: Initialized, Role: LEARNER
I20240501 07:59:08.872169 9805 consensus_queue.cc:260] T ec2d3ccf0937430f947f3e2bde0506c5 P 13d05ebdca8c4f489c597ce550dd773d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { replace: true } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: true } }
I20240501 07:59:08.872501 9805 ts_tablet_manager.cc:1398] T ec2d3ccf0937430f947f3e2bde0506c5 P 13d05ebdca8c4f489c597ce550dd773d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:08.873831 6335 tablet_copy_service.cc:342] P 8b8b3ad3aa8f4053b2167dd550f2086f: Request end of tablet copy session 13d05ebdca8c4f489c597ce550dd773d-ec2d3ccf0937430f947f3e2bde0506c5 received from {username='slave'} at 127.0.99.134:55617
I20240501 07:59:08.873937 6335 tablet_copy_service.cc:433] P 8b8b3ad3aa8f4053b2167dd550f2086f: ending tablet copy session 13d05ebdca8c4f489c597ce550dd773d-ec2d3ccf0937430f947f3e2bde0506c5 on tablet ec2d3ccf0937430f947f3e2bde0506c5 with peer 13d05ebdca8c4f489c597ce550dd773d
I20240501 07:59:08.875480 9740 tablet_bootstrap.cc:492] T 404bb81b9c404dd288d2f8160a4273da P 83f20e84af4c40d09a4804b6b26e697f: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:08.875763 9740 tablet_bootstrap.cc:492] T 404bb81b9c404dd288d2f8160a4273da P 83f20e84af4c40d09a4804b6b26e697f: Bootstrap complete.
I20240501 07:59:08.875934 9740 ts_tablet_manager.cc:1367] T 404bb81b9c404dd288d2f8160a4273da P 83f20e84af4c40d09a4804b6b26e697f: Time spent bootstrapping tablet: real 0.081s user 0.028s sys 0.000s
I20240501 07:59:08.876307 9740 raft_consensus.cc:348] T 404bb81b9c404dd288d2f8160a4273da P 83f20e84af4c40d09a4804b6b26e697f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: true } }
I20240501 07:59:08.876415 9740 raft_consensus.cc:729] T 404bb81b9c404dd288d2f8160a4273da P 83f20e84af4c40d09a4804b6b26e697f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 83f20e84af4c40d09a4804b6b26e697f, State: Initialized, Role: LEARNER
I20240501 07:59:08.876530 9740 consensus_queue.cc:260] T 404bb81b9c404dd288d2f8160a4273da P 83f20e84af4c40d09a4804b6b26e697f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: true } }
I20240501 07:59:08.876893 9740 ts_tablet_manager.cc:1398] T 404bb81b9c404dd288d2f8160a4273da P 83f20e84af4c40d09a4804b6b26e697f: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240501 07:59:08.877159 9854 tablet_copy_client.cc:523] T 6e2ac8c8857d45b88d3b51147d2609aa P 83f20e84af4c40d09a4804b6b26e697f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:08.877452 6727 tablet_copy_service.cc:342] P 1f9cb72be98941b5a8e504cbe34132b9: Request end of tablet copy session 83f20e84af4c40d09a4804b6b26e697f-404bb81b9c404dd288d2f8160a4273da received from {username='slave'} at 127.0.99.136:55471
I20240501 07:59:08.877532 6727 tablet_copy_service.cc:433] P 1f9cb72be98941b5a8e504cbe34132b9: ending tablet copy session 83f20e84af4c40d09a4804b6b26e697f-404bb81b9c404dd288d2f8160a4273da on tablet 404bb81b9c404dd288d2f8160a4273da with peer 83f20e84af4c40d09a4804b6b26e697f
I20240501 07:59:08.878461 9795 tablet_copy_client.cc:523] T 5ee738571af740c1b494d4f17413d1d8 P 74077b61d6f54342bf9736b4a9f2eade: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:08.878688 9854 tablet_bootstrap.cc:492] T 6e2ac8c8857d45b88d3b51147d2609aa P 83f20e84af4c40d09a4804b6b26e697f: Bootstrap starting.
I20240501 07:59:08.879840 9795 tablet_bootstrap.cc:492] T 5ee738571af740c1b494d4f17413d1d8 P 74077b61d6f54342bf9736b4a9f2eade: Bootstrap starting.
I20240501 07:59:08.885118 9805 ts_tablet_manager.cc:902] T ff9dc384dbac47dc9795617c64688943 P 13d05ebdca8c4f489c597ce550dd773d: Initiating tablet copy from peer 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131:34471)
I20240501 07:59:08.886821 9805 tablet_copy_client.cc:308] T ff9dc384dbac47dc9795617c64688943 P 13d05ebdca8c4f489c597ce550dd773d: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:34471
I20240501 07:59:08.888695 6166 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55950:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 07:59:08.888839 6166 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55950:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 07:59:08.892899 6597 tablet_copy_service.cc:140] P 3b164cbbc00f4e8eb3254ddd349a3ff9: Received BeginTabletCopySession request for tablet ff9dc384dbac47dc9795617c64688943 from peer 13d05ebdca8c4f489c597ce550dd773d ({username='slave'} at 127.0.99.134:42357)
I20240501 07:59:08.893041 6597 tablet_copy_service.cc:161] P 3b164cbbc00f4e8eb3254ddd349a3ff9: Beginning new tablet copy session on tablet ff9dc384dbac47dc9795617c64688943 from peer 13d05ebdca8c4f489c597ce550dd773d at {username='slave'} at 127.0.99.134:42357: session id = 13d05ebdca8c4f489c597ce550dd773d-ff9dc384dbac47dc9795617c64688943
I20240501 07:59:08.894029 6597 tablet_copy_source_session.cc:205] T ff9dc384dbac47dc9795617c64688943 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:08.894213 6166 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 0bc706e85ce74cef97a4827ea4d4aa65: Sending DeleteTablet for 3 replicas of tablet 4cdabbbfd1df4a5a9683e3c8e569d624
I20240501 07:59:08.894963 8905 tablet_service.cc:1514] Processing DeleteTablet for tablet 4cdabbbfd1df4a5a9683e3c8e569d624 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:08 UTC) from {username='slave'} at 127.0.0.1:47202
I20240501 07:59:08.894935 8774 tablet_service.cc:1514] Processing DeleteTablet for tablet 4cdabbbfd1df4a5a9683e3c8e569d624 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:08 UTC) from {username='slave'} at 127.0.0.1:43136
I20240501 07:59:08.895328 9167 tablet_service.cc:1514] Processing DeleteTablet for tablet 4cdabbbfd1df4a5a9683e3c8e569d624 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:08 UTC) from {username='slave'} at 127.0.0.1:33594
I20240501 07:59:08.895560 9805 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff9dc384dbac47dc9795617c64688943. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:08.901947 9784 ts_tablet_manager.cc:902] T 782e37a4ebb34a68b461a74115ae8703 P a505e7cb133c4392bfda8a4a20874ae9: Initiating tablet copy from peer 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130:32941)
I20240501 07:59:08.902474 9784 tablet_copy_client.cc:308] T 782e37a4ebb34a68b461a74115ae8703 P a505e7cb133c4392bfda8a4a20874ae9: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:32941
I20240501 07:59:08.904407 9805 tablet_copy_client.cc:783] T ff9dc384dbac47dc9795617c64688943 P 13d05ebdca8c4f489c597ce550dd773d: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:08.904592 9805 tablet_copy_client.cc:647] T ff9dc384dbac47dc9795617c64688943 P 13d05ebdca8c4f489c597ce550dd773d: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:08.907171 6466 tablet_copy_service.cc:140] P 8315d22c56f7439a9e1fc82f153f018c: Received BeginTabletCopySession request for tablet 782e37a4ebb34a68b461a74115ae8703 from peer a505e7cb133c4392bfda8a4a20874ae9 ({username='slave'} at 127.0.99.135:36631)
I20240501 07:59:08.907294 6466 tablet_copy_service.cc:161] P 8315d22c56f7439a9e1fc82f153f018c: Beginning new tablet copy session on tablet 782e37a4ebb34a68b461a74115ae8703 from peer a505e7cb133c4392bfda8a4a20874ae9 at {username='slave'} at 127.0.99.135:36631: session id = a505e7cb133c4392bfda8a4a20874ae9-782e37a4ebb34a68b461a74115ae8703
I20240501 07:59:08.908227 6466 tablet_copy_source_session.cc:205] T 782e37a4ebb34a68b461a74115ae8703 P 8315d22c56f7439a9e1fc82f153f018c: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:08.909155 9784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 782e37a4ebb34a68b461a74115ae8703. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:08.911440 9784 tablet_copy_client.cc:783] T 782e37a4ebb34a68b461a74115ae8703 P a505e7cb133c4392bfda8a4a20874ae9: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:08.911595 9784 tablet_copy_client.cc:647] T 782e37a4ebb34a68b461a74115ae8703 P a505e7cb133c4392bfda8a4a20874ae9: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:08.913801 9784 tablet_copy_client.cc:523] T 782e37a4ebb34a68b461a74115ae8703 P a505e7cb133c4392bfda8a4a20874ae9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:08.915155 9784 tablet_bootstrap.cc:492] T 782e37a4ebb34a68b461a74115ae8703 P a505e7cb133c4392bfda8a4a20874ae9: Bootstrap starting.
I20240501 07:59:08.916288 9805 tablet_copy_client.cc:523] T ff9dc384dbac47dc9795617c64688943 P 13d05ebdca8c4f489c597ce550dd773d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:08.917896 9805 tablet_bootstrap.cc:492] T ff9dc384dbac47dc9795617c64688943 P 13d05ebdca8c4f489c597ce550dd773d: Bootstrap starting.
I20240501 07:59:08.921398 9795 tablet_bootstrap.cc:492] T 5ee738571af740c1b494d4f17413d1d8 P 74077b61d6f54342bf9736b4a9f2eade: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:08.921823 9795 tablet_bootstrap.cc:492] T 5ee738571af740c1b494d4f17413d1d8 P 74077b61d6f54342bf9736b4a9f2eade: Bootstrap complete.
I20240501 07:59:08.922032 9795 ts_tablet_manager.cc:1367] T 5ee738571af740c1b494d4f17413d1d8 P 74077b61d6f54342bf9736b4a9f2eade: Time spent bootstrapping tablet: real 0.042s user 0.026s sys 0.000s
I20240501 07:59:08.922473 9893 tablet_replica.cc:330] T 4cdabbbfd1df4a5a9683e3c8e569d624 P 13d05ebdca8c4f489c597ce550dd773d: stopping tablet replica
I20240501 07:59:08.922503 9795 raft_consensus.cc:348] T 5ee738571af740c1b494d4f17413d1d8 P 74077b61d6f54342bf9736b4a9f2eade [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { replace: true } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: true } }
I20240501 07:59:08.922796 9795 raft_consensus.cc:729] T 5ee738571af740c1b494d4f17413d1d8 P 74077b61d6f54342bf9736b4a9f2eade [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 74077b61d6f54342bf9736b4a9f2eade, State: Initialized, Role: LEARNER
I20240501 07:59:08.922794 9893 raft_consensus.cc:2229] T 4cdabbbfd1df4a5a9683e3c8e569d624 P 13d05ebdca8c4f489c597ce550dd773d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:08.922966 9893 raft_consensus.cc:2258] T 4cdabbbfd1df4a5a9683e3c8e569d624 P 13d05ebdca8c4f489c597ce550dd773d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:08.922947 9795 consensus_queue.cc:260] T 5ee738571af740c1b494d4f17413d1d8 P 74077b61d6f54342bf9736b4a9f2eade [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { replace: true } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: true } }
I20240501 07:59:08.924048 9893 ts_tablet_manager.cc:1875] T 4cdabbbfd1df4a5a9683e3c8e569d624 P 13d05ebdca8c4f489c597ce550dd773d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:59:08.925117 9795 ts_tablet_manager.cc:1398] T 5ee738571af740c1b494d4f17413d1d8 P 74077b61d6f54342bf9736b4a9f2eade: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 07:59:08.925132 9895 tablet_replica.cc:330] T 4cdabbbfd1df4a5a9683e3c8e569d624 P 83f20e84af4c40d09a4804b6b26e697f: stopping tablet replica
I20240501 07:59:08.925259 9895 raft_consensus.cc:2229] T 4cdabbbfd1df4a5a9683e3c8e569d624 P 83f20e84af4c40d09a4804b6b26e697f [term 1 LEADER]: Raft consensus shutting down.
I20240501 07:59:08.925580 9895 raft_consensus.cc:2258] T 4cdabbbfd1df4a5a9683e3c8e569d624 P 83f20e84af4c40d09a4804b6b26e697f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:08.927091 9893 ts_tablet_manager.cc:1888] T 4cdabbbfd1df4a5a9683e3c8e569d624 P 13d05ebdca8c4f489c597ce550dd773d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:59:08.927292 6335 tablet_copy_service.cc:342] P 8b8b3ad3aa8f4053b2167dd550f2086f: Request end of tablet copy session 74077b61d6f54342bf9736b4a9f2eade-5ee738571af740c1b494d4f17413d1d8 received from {username='slave'} at 127.0.99.133:50111
I20240501 07:59:08.927397 6335 tablet_copy_service.cc:433] P 8b8b3ad3aa8f4053b2167dd550f2086f: ending tablet copy session 74077b61d6f54342bf9736b4a9f2eade-5ee738571af740c1b494d4f17413d1d8 on tablet 5ee738571af740c1b494d4f17413d1d8 with peer 74077b61d6f54342bf9736b4a9f2eade
I20240501 07:59:08.927460 9895 ts_tablet_manager.cc:1875] T 4cdabbbfd1df4a5a9683e3c8e569d624 P 83f20e84af4c40d09a4804b6b26e697f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:59:08.927649 9894 tablet_replica.cc:330] T 4cdabbbfd1df4a5a9683e3c8e569d624 P 74077b61d6f54342bf9736b4a9f2eade: stopping tablet replica
I20240501 07:59:08.930320 9895 ts_tablet_manager.cc:1888] T 4cdabbbfd1df4a5a9683e3c8e569d624 P 83f20e84af4c40d09a4804b6b26e697f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:59:08.930456 9895 log.cc:1196] T 4cdabbbfd1df4a5a9683e3c8e569d624 P 83f20e84af4c40d09a4804b6b26e697f: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/4cdabbbfd1df4a5a9683e3c8e569d624
I20240501 07:59:08.930783 9895 ts_tablet_manager.cc:1909] T 4cdabbbfd1df4a5a9683e3c8e569d624 P 83f20e84af4c40d09a4804b6b26e697f: Deleting consensus metadata
I20240501 07:59:08.931668 6158 catalog_manager.cc:4830] TS 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): tablet 4cdabbbfd1df4a5a9683e3c8e569d624 (table rebalancer_dynamic_table_new_name [id=5a2c4b25a3944383893b9fd726c21cad]) successfully deleted
I20240501 07:59:08.932555 9790 tablet_bootstrap.cc:492] T 3b9b71d44f754353b2321e8c36dd4dfd P a505e7cb133c4392bfda8a4a20874ae9: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:08.933048 9894 raft_consensus.cc:2229] T 4cdabbbfd1df4a5a9683e3c8e569d624 P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:08.933274 9894 raft_consensus.cc:2258] T 4cdabbbfd1df4a5a9683e3c8e569d624 P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:08.934417 9894 ts_tablet_manager.cc:1875] T 4cdabbbfd1df4a5a9683e3c8e569d624 P 74077b61d6f54342bf9736b4a9f2eade: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:59:08.936201 9893 log.cc:1196] T 4cdabbbfd1df4a5a9683e3c8e569d624 P 13d05ebdca8c4f489c597ce550dd773d: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/4cdabbbfd1df4a5a9683e3c8e569d624
I20240501 07:59:08.936555 9893 ts_tablet_manager.cc:1909] T 4cdabbbfd1df4a5a9683e3c8e569d624 P 13d05ebdca8c4f489c597ce550dd773d: Deleting consensus metadata
I20240501 07:59:08.937029 9894 ts_tablet_manager.cc:1888] T 4cdabbbfd1df4a5a9683e3c8e569d624 P 74077b61d6f54342bf9736b4a9f2eade: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:59:08.937156 9894 log.cc:1196] T 4cdabbbfd1df4a5a9683e3c8e569d624 P 74077b61d6f54342bf9736b4a9f2eade: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/4cdabbbfd1df4a5a9683e3c8e569d624
I20240501 07:59:08.937484 9894 ts_tablet_manager.cc:1909] T 4cdabbbfd1df4a5a9683e3c8e569d624 P 74077b61d6f54342bf9736b4a9f2eade: Deleting consensus metadata
I20240501 07:59:08.937852 6159 catalog_manager.cc:4830] TS 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134:40141): tablet 4cdabbbfd1df4a5a9683e3c8e569d624 (table rebalancer_dynamic_table_new_name [id=5a2c4b25a3944383893b9fd726c21cad]) successfully deleted
I20240501 07:59:08.938622 6157 catalog_manager.cc:4830] TS 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133:39859): tablet 4cdabbbfd1df4a5a9683e3c8e569d624 (table rebalancer_dynamic_table_new_name [id=5a2c4b25a3944383893b9fd726c21cad]) successfully deleted
I20240501 07:59:08.938804 9790 tablet_bootstrap.cc:492] T 3b9b71d44f754353b2321e8c36dd4dfd P a505e7cb133c4392bfda8a4a20874ae9: Bootstrap complete.
I20240501 07:59:08.939018 9790 ts_tablet_manager.cc:1367] T 3b9b71d44f754353b2321e8c36dd4dfd P a505e7cb133c4392bfda8a4a20874ae9: Time spent bootstrapping tablet: real 0.094s user 0.023s sys 0.004s
I20240501 07:59:08.939549 9790 raft_consensus.cc:348] T 3b9b71d44f754353b2321e8c36dd4dfd P a505e7cb133c4392bfda8a4a20874ae9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: true } }
I20240501 07:59:08.939742 9790 raft_consensus.cc:729] T 3b9b71d44f754353b2321e8c36dd4dfd P a505e7cb133c4392bfda8a4a20874ae9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a505e7cb133c4392bfda8a4a20874ae9, State: Initialized, Role: LEARNER
I20240501 07:59:08.939899 9790 consensus_queue.cc:260] T 3b9b71d44f754353b2321e8c36dd4dfd P a505e7cb133c4392bfda8a4a20874ae9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: true } }
I20240501 07:59:08.940865 9790 ts_tablet_manager.cc:1398] T 3b9b71d44f754353b2321e8c36dd4dfd P a505e7cb133c4392bfda8a4a20874ae9: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:59:08.941464 6727 tablet_copy_service.cc:342] P 1f9cb72be98941b5a8e504cbe34132b9: Request end of tablet copy session a505e7cb133c4392bfda8a4a20874ae9-3b9b71d44f754353b2321e8c36dd4dfd received from {username='slave'} at 127.0.99.135:56663
I20240501 07:59:08.941543 6727 tablet_copy_service.cc:433] P 1f9cb72be98941b5a8e504cbe34132b9: ending tablet copy session a505e7cb133c4392bfda8a4a20874ae9-3b9b71d44f754353b2321e8c36dd4dfd on tablet 3b9b71d44f754353b2321e8c36dd4dfd with peer a505e7cb133c4392bfda8a4a20874ae9
I20240501 07:59:08.949335 9846 tablet_bootstrap.cc:492] T 187940ba79f546b1ad79e7ec1de76442 P 13d05ebdca8c4f489c597ce550dd773d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:08.949688 9846 tablet_bootstrap.cc:492] T 187940ba79f546b1ad79e7ec1de76442 P 13d05ebdca8c4f489c597ce550dd773d: Bootstrap complete.
I20240501 07:59:08.949870 9846 ts_tablet_manager.cc:1367] T 187940ba79f546b1ad79e7ec1de76442 P 13d05ebdca8c4f489c597ce550dd773d: Time spent bootstrapping tablet: real 0.083s user 0.026s sys 0.002s
I20240501 07:59:08.950343 9846 raft_consensus.cc:348] T 187940ba79f546b1ad79e7ec1de76442 P 13d05ebdca8c4f489c597ce550dd773d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { replace: true } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: true } }
I20240501 07:59:08.950482 9846 raft_consensus.cc:729] T 187940ba79f546b1ad79e7ec1de76442 P 13d05ebdca8c4f489c597ce550dd773d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 13d05ebdca8c4f489c597ce550dd773d, State: Initialized, Role: LEARNER
I20240501 07:59:08.950582 9846 consensus_queue.cc:260] T 187940ba79f546b1ad79e7ec1de76442 P 13d05ebdca8c4f489c597ce550dd773d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { replace: true } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: true } }
I20240501 07:59:08.951231 9846 ts_tablet_manager.cc:1398] T 187940ba79f546b1ad79e7ec1de76442 P 13d05ebdca8c4f489c597ce550dd773d: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240501 07:59:08.951545 9805 tablet_bootstrap.cc:492] T ff9dc384dbac47dc9795617c64688943 P 13d05ebdca8c4f489c597ce550dd773d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:08.951795 9805 tablet_bootstrap.cc:492] T ff9dc384dbac47dc9795617c64688943 P 13d05ebdca8c4f489c597ce550dd773d: Bootstrap complete.
I20240501 07:59:08.951875 6727 tablet_copy_service.cc:342] P 1f9cb72be98941b5a8e504cbe34132b9: Request end of tablet copy session 13d05ebdca8c4f489c597ce550dd773d-187940ba79f546b1ad79e7ec1de76442 received from {username='slave'} at 127.0.99.134:44489
I20240501 07:59:08.951977 6727 tablet_copy_service.cc:433] P 1f9cb72be98941b5a8e504cbe34132b9: ending tablet copy session 13d05ebdca8c4f489c597ce550dd773d-187940ba79f546b1ad79e7ec1de76442 on tablet 187940ba79f546b1ad79e7ec1de76442 with peer 13d05ebdca8c4f489c597ce550dd773d
I20240501 07:59:08.951973 9805 ts_tablet_manager.cc:1367] T ff9dc384dbac47dc9795617c64688943 P 13d05ebdca8c4f489c597ce550dd773d: Time spent bootstrapping tablet: real 0.034s user 0.019s sys 0.003s
I20240501 07:59:08.953060 9795 ts_tablet_manager.cc:902] T c3a9036ccfee45498bca9681518d3699 P 74077b61d6f54342bf9736b4a9f2eade: Initiating tablet copy from peer 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132:36773)
I20240501 07:59:08.953492 9795 tablet_copy_client.cc:308] T c3a9036ccfee45498bca9681518d3699 P 74077b61d6f54342bf9736b4a9f2eade: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:36773
I20240501 07:59:08.956220 9860 tablet_bootstrap.cc:492] T bab34f083c8045e4a6df373db5ebd54b P 13d05ebdca8c4f489c597ce550dd773d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:08.956561 9860 tablet_bootstrap.cc:492] T bab34f083c8045e4a6df373db5ebd54b P 13d05ebdca8c4f489c597ce550dd773d: Bootstrap complete.
I20240501 07:59:08.956769 9860 ts_tablet_manager.cc:1367] T bab34f083c8045e4a6df373db5ebd54b P 13d05ebdca8c4f489c597ce550dd773d: Time spent bootstrapping tablet: real 0.089s user 0.025s sys 0.004s
I20240501 07:59:08.957250 9860 raft_consensus.cc:348] T bab34f083c8045e4a6df373db5ebd54b P 13d05ebdca8c4f489c597ce550dd773d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: true } }
I20240501 07:59:08.957455 9860 raft_consensus.cc:729] T bab34f083c8045e4a6df373db5ebd54b P 13d05ebdca8c4f489c597ce550dd773d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 13d05ebdca8c4f489c597ce550dd773d, State: Initialized, Role: LEARNER
I20240501 07:59:08.957616 9860 consensus_queue.cc:260] T bab34f083c8045e4a6df373db5ebd54b P 13d05ebdca8c4f489c597ce550dd773d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: true } }
I20240501 07:59:08.958019 9860 ts_tablet_manager.cc:1398] T bab34f083c8045e4a6df373db5ebd54b P 13d05ebdca8c4f489c597ce550dd773d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:08.959195 6727 tablet_copy_service.cc:140] P 1f9cb72be98941b5a8e504cbe34132b9: Received BeginTabletCopySession request for tablet c3a9036ccfee45498bca9681518d3699 from peer 74077b61d6f54342bf9736b4a9f2eade ({username='slave'} at 127.0.99.133:46973)
I20240501 07:59:08.959319 6727 tablet_copy_service.cc:161] P 1f9cb72be98941b5a8e504cbe34132b9: Beginning new tablet copy session on tablet c3a9036ccfee45498bca9681518d3699 from peer 74077b61d6f54342bf9736b4a9f2eade at {username='slave'} at 127.0.99.133:46973: session id = 74077b61d6f54342bf9736b4a9f2eade-c3a9036ccfee45498bca9681518d3699
I20240501 07:59:08.960263 6727 tablet_copy_source_session.cc:205] T c3a9036ccfee45498bca9681518d3699 P 1f9cb72be98941b5a8e504cbe34132b9: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:08.961338 9795 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3a9036ccfee45498bca9681518d3699. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:08.963446 9795 tablet_copy_client.cc:783] T c3a9036ccfee45498bca9681518d3699 P 74077b61d6f54342bf9736b4a9f2eade: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:08.963594 9795 tablet_copy_client.cc:647] T c3a9036ccfee45498bca9681518d3699 P 74077b61d6f54342bf9736b4a9f2eade: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:08.964740 6466 tablet_copy_service.cc:342] P 8315d22c56f7439a9e1fc82f153f018c: Request end of tablet copy session 13d05ebdca8c4f489c597ce550dd773d-bab34f083c8045e4a6df373db5ebd54b received from {username='slave'} at 127.0.99.134:41693
I20240501 07:59:08.964820 6466 tablet_copy_service.cc:433] P 8315d22c56f7439a9e1fc82f153f018c: ending tablet copy session 13d05ebdca8c4f489c597ce550dd773d-bab34f083c8045e4a6df373db5ebd54b on tablet bab34f083c8045e4a6df373db5ebd54b with peer 13d05ebdca8c4f489c597ce550dd773d
I20240501 07:59:08.965548 9778 ts_tablet_manager.cc:902] T b7d57b7fd5234f50afbc859bc997f3e1 P 74077b61d6f54342bf9736b4a9f2eade: Initiating tablet copy from peer 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131:34471)
I20240501 07:59:08.966030 9778 tablet_copy_client.cc:308] T b7d57b7fd5234f50afbc859bc997f3e1 P 74077b61d6f54342bf9736b4a9f2eade: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:34471
I20240501 07:59:08.966579 6597 tablet_copy_service.cc:140] P 3b164cbbc00f4e8eb3254ddd349a3ff9: Received BeginTabletCopySession request for tablet b7d57b7fd5234f50afbc859bc997f3e1 from peer 74077b61d6f54342bf9736b4a9f2eade ({username='slave'} at 127.0.99.133:57559)
I20240501 07:59:08.966691 6597 tablet_copy_service.cc:161] P 3b164cbbc00f4e8eb3254ddd349a3ff9: Beginning new tablet copy session on tablet b7d57b7fd5234f50afbc859bc997f3e1 from peer 74077b61d6f54342bf9736b4a9f2eade at {username='slave'} at 127.0.99.133:57559: session id = 74077b61d6f54342bf9736b4a9f2eade-b7d57b7fd5234f50afbc859bc997f3e1
I20240501 07:59:08.966943 9790 ts_tablet_manager.cc:902] T 8454f24b0c28451b8ee51be39ca3e07f P a505e7cb133c4392bfda8a4a20874ae9: Initiating tablet copy from peer 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132:36773)
I20240501 07:59:08.967348 9790 tablet_copy_client.cc:308] T 8454f24b0c28451b8ee51be39ca3e07f P a505e7cb133c4392bfda8a4a20874ae9: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:36773
I20240501 07:59:08.967710 6597 tablet_copy_source_session.cc:205] T b7d57b7fd5234f50afbc859bc997f3e1 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:08.968684 9778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7d57b7fd5234f50afbc859bc997f3e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:08.970752 9778 tablet_copy_client.cc:783] T b7d57b7fd5234f50afbc859bc997f3e1 P 74077b61d6f54342bf9736b4a9f2eade: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:08.970893 9778 tablet_copy_client.cc:647] T b7d57b7fd5234f50afbc859bc997f3e1 P 74077b61d6f54342bf9736b4a9f2eade: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:08.971362 9784 tablet_bootstrap.cc:492] T 782e37a4ebb34a68b461a74115ae8703 P a505e7cb133c4392bfda8a4a20874ae9: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:08.971778 9784 tablet_bootstrap.cc:492] T 782e37a4ebb34a68b461a74115ae8703 P a505e7cb133c4392bfda8a4a20874ae9: Bootstrap complete.
I20240501 07:59:08.972010 9784 ts_tablet_manager.cc:1367] T 782e37a4ebb34a68b461a74115ae8703 P a505e7cb133c4392bfda8a4a20874ae9: Time spent bootstrapping tablet: real 0.057s user 0.027s sys 0.004s
I20240501 07:59:08.972339 6728 tablet_copy_service.cc:140] P 1f9cb72be98941b5a8e504cbe34132b9: Received BeginTabletCopySession request for tablet 8454f24b0c28451b8ee51be39ca3e07f from peer a505e7cb133c4392bfda8a4a20874ae9 ({username='slave'} at 127.0.99.135:56663)
I20240501 07:59:08.972461 6728 tablet_copy_service.cc:161] P 1f9cb72be98941b5a8e504cbe34132b9: Beginning new tablet copy session on tablet 8454f24b0c28451b8ee51be39ca3e07f from peer a505e7cb133c4392bfda8a4a20874ae9 at {username='slave'} at 127.0.99.135:56663: session id = a505e7cb133c4392bfda8a4a20874ae9-8454f24b0c28451b8ee51be39ca3e07f
I20240501 07:59:08.972703 9778 tablet_copy_client.cc:523] T b7d57b7fd5234f50afbc859bc997f3e1 P 74077b61d6f54342bf9736b4a9f2eade: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:08.972697 9784 raft_consensus.cc:348] T 782e37a4ebb34a68b461a74115ae8703 P a505e7cb133c4392bfda8a4a20874ae9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: true } }
I20240501 07:59:08.972890 9784 raft_consensus.cc:729] T 782e37a4ebb34a68b461a74115ae8703 P a505e7cb133c4392bfda8a4a20874ae9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a505e7cb133c4392bfda8a4a20874ae9, State: Initialized, Role: LEARNER
I20240501 07:59:08.973055 9784 consensus_queue.cc:260] T 782e37a4ebb34a68b461a74115ae8703 P a505e7cb133c4392bfda8a4a20874ae9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: true } }
I20240501 07:59:08.973392 6728 tablet_copy_source_session.cc:205] T 8454f24b0c28451b8ee51be39ca3e07f P 1f9cb72be98941b5a8e504cbe34132b9: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:08.973466 9784 ts_tablet_manager.cc:1398] T 782e37a4ebb34a68b461a74115ae8703 P a505e7cb133c4392bfda8a4a20874ae9: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 07:59:08.973987 6466 tablet_copy_service.cc:342] P 8315d22c56f7439a9e1fc82f153f018c: Request end of tablet copy session a505e7cb133c4392bfda8a4a20874ae9-782e37a4ebb34a68b461a74115ae8703 received from {username='slave'} at 127.0.99.135:36631
I20240501 07:59:08.974054 6466 tablet_copy_service.cc:433] P 8315d22c56f7439a9e1fc82f153f018c: ending tablet copy session a505e7cb133c4392bfda8a4a20874ae9-782e37a4ebb34a68b461a74115ae8703 on tablet 782e37a4ebb34a68b461a74115ae8703 with peer a505e7cb133c4392bfda8a4a20874ae9
I20240501 07:59:08.974170 9805 raft_consensus.cc:348] T ff9dc384dbac47dc9795617c64688943 P 13d05ebdca8c4f489c597ce550dd773d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: true } }
I20240501 07:59:08.974536 9805 raft_consensus.cc:729] T ff9dc384dbac47dc9795617c64688943 P 13d05ebdca8c4f489c597ce550dd773d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 13d05ebdca8c4f489c597ce550dd773d, State: Initialized, Role: LEARNER
I20240501 07:59:08.974676 9790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8454f24b0c28451b8ee51be39ca3e07f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:08.974682 9805 consensus_queue.cc:260] T ff9dc384dbac47dc9795617c64688943 P 13d05ebdca8c4f489c597ce550dd773d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: true } }
I20240501 07:59:08.975097 9805 ts_tablet_manager.cc:1398] T ff9dc384dbac47dc9795617c64688943 P 13d05ebdca8c4f489c597ce550dd773d: Time spent starting tablet: real 0.023s user 0.003s sys 0.001s
I20240501 07:59:08.975503 9795 tablet_copy_client.cc:523] T c3a9036ccfee45498bca9681518d3699 P 74077b61d6f54342bf9736b4a9f2eade: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:08.976748 9795 tablet_bootstrap.cc:492] T c3a9036ccfee45498bca9681518d3699 P 74077b61d6f54342bf9736b4a9f2eade: Bootstrap starting.
I20240501 07:59:08.976820 9790 tablet_copy_client.cc:783] T 8454f24b0c28451b8ee51be39ca3e07f P a505e7cb133c4392bfda8a4a20874ae9: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:08.976953 9790 tablet_copy_client.cc:647] T 8454f24b0c28451b8ee51be39ca3e07f P a505e7cb133c4392bfda8a4a20874ae9: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:08.977912 6597 tablet_copy_service.cc:342] P 3b164cbbc00f4e8eb3254ddd349a3ff9: Request end of tablet copy session 13d05ebdca8c4f489c597ce550dd773d-ff9dc384dbac47dc9795617c64688943 received from {username='slave'} at 127.0.99.134:42357
I20240501 07:59:08.977990 6597 tablet_copy_service.cc:433] P 3b164cbbc00f4e8eb3254ddd349a3ff9: ending tablet copy session 13d05ebdca8c4f489c597ce550dd773d-ff9dc384dbac47dc9795617c64688943 on tablet ff9dc384dbac47dc9795617c64688943 with peer 13d05ebdca8c4f489c597ce550dd773d
I20240501 07:59:08.979259 9790 tablet_copy_client.cc:523] T 8454f24b0c28451b8ee51be39ca3e07f P a505e7cb133c4392bfda8a4a20874ae9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:08.979822 9778 tablet_bootstrap.cc:492] T b7d57b7fd5234f50afbc859bc997f3e1 P 74077b61d6f54342bf9736b4a9f2eade: Bootstrap starting.
I20240501 07:59:08.982280 9790 tablet_bootstrap.cc:492] T 8454f24b0c28451b8ee51be39ca3e07f P a505e7cb133c4392bfda8a4a20874ae9: Bootstrap starting.
I20240501 07:59:08.997895 6172 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55950:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 07:59:09.000844 9116 consensus_queue.cc:1035] T cfcc814f4d2d488ba745ecaf0cc6d61a P 8315d22c56f7439a9e1fc82f153f018c [LEADER]: Connected to new peer: Peer: permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:09.005542 9854 tablet_bootstrap.cc:492] T 6e2ac8c8857d45b88d3b51147d2609aa P 83f20e84af4c40d09a4804b6b26e697f: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:09.007130 8774 tablet_service.cc:1467] Processing CreateTablet for tablet 9a17ba3dfb234636bcf63c3340aaab9d (DEFAULT_TABLE table=rebalancer_dynamic_table [id=81aa8832269b4874bfec689c1e98d696]), partition=
I20240501 07:59:09.007392 8774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a17ba3dfb234636bcf63c3340aaab9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:09.007952 9795 tablet_bootstrap.cc:492] T c3a9036ccfee45498bca9681518d3699 P 74077b61d6f54342bf9736b4a9f2eade: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:09.008220 9795 tablet_bootstrap.cc:492] T c3a9036ccfee45498bca9681518d3699 P 74077b61d6f54342bf9736b4a9f2eade: Bootstrap complete.
I20240501 07:59:09.008385 9795 ts_tablet_manager.cc:1367] T c3a9036ccfee45498bca9681518d3699 P 74077b61d6f54342bf9736b4a9f2eade: Time spent bootstrapping tablet: real 0.032s user 0.027s sys 0.000s
I20240501 07:59:09.008839 9795 raft_consensus.cc:348] T c3a9036ccfee45498bca9681518d3699 P 74077b61d6f54342bf9736b4a9f2eade [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { replace: true } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: true } }
I20240501 07:59:09.008985 9795 raft_consensus.cc:729] T c3a9036ccfee45498bca9681518d3699 P 74077b61d6f54342bf9736b4a9f2eade [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 74077b61d6f54342bf9736b4a9f2eade, State: Initialized, Role: LEARNER
I20240501 07:59:09.009085 9795 consensus_queue.cc:260] T c3a9036ccfee45498bca9681518d3699 P 74077b61d6f54342bf9736b4a9f2eade [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { replace: true } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: true } }
I20240501 07:59:09.009680 8905 tablet_service.cc:1467] Processing CreateTablet for tablet 9a17ba3dfb234636bcf63c3340aaab9d (DEFAULT_TABLE table=rebalancer_dynamic_table [id=81aa8832269b4874bfec689c1e98d696]), partition=
I20240501 07:59:09.009719 9392 tablet_bootstrap.cc:492] T 9a17ba3dfb234636bcf63c3340aaab9d P 74077b61d6f54342bf9736b4a9f2eade: Bootstrap starting.
I20240501 07:59:09.009825 9795 ts_tablet_manager.cc:1398] T c3a9036ccfee45498bca9681518d3699 P 74077b61d6f54342bf9736b4a9f2eade: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240501 07:59:09.009919 8905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a17ba3dfb234636bcf63c3340aaab9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:09.010588 6728 tablet_copy_service.cc:342] P 1f9cb72be98941b5a8e504cbe34132b9: Request end of tablet copy session 74077b61d6f54342bf9736b4a9f2eade-c3a9036ccfee45498bca9681518d3699 received from {username='slave'} at 127.0.99.133:46973
I20240501 07:59:09.010656 9392 tablet_bootstrap.cc:654] T 9a17ba3dfb234636bcf63c3340aaab9d P 74077b61d6f54342bf9736b4a9f2eade: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:09.010681 6728 tablet_copy_service.cc:433] P 1f9cb72be98941b5a8e504cbe34132b9: ending tablet copy session 74077b61d6f54342bf9736b4a9f2eade-c3a9036ccfee45498bca9681518d3699 on tablet c3a9036ccfee45498bca9681518d3699 with peer 74077b61d6f54342bf9736b4a9f2eade
I20240501 07:59:09.018635 9392 tablet_bootstrap.cc:492] T 9a17ba3dfb234636bcf63c3340aaab9d P 74077b61d6f54342bf9736b4a9f2eade: No bootstrap required, opened a new log
I20240501 07:59:09.018745 9392 ts_tablet_manager.cc:1367] T 9a17ba3dfb234636bcf63c3340aaab9d P 74077b61d6f54342bf9736b4a9f2eade: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.002s
I20240501 07:59:09.019202 9392 raft_consensus.cc:348] T 9a17ba3dfb234636bcf63c3340aaab9d P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } }
I20240501 07:59:09.019309 9392 raft_consensus.cc:374] T 9a17ba3dfb234636bcf63c3340aaab9d P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:09.019340 9392 raft_consensus.cc:729] T 9a17ba3dfb234636bcf63c3340aaab9d P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74077b61d6f54342bf9736b4a9f2eade, State: Initialized, Role: FOLLOWER
I20240501 07:59:09.019466 9392 consensus_queue.cc:260] T 9a17ba3dfb234636bcf63c3340aaab9d P 74077b61d6f54342bf9736b4a9f2eade [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } }
I20240501 07:59:09.019730 9392 ts_tablet_manager.cc:1398] T 9a17ba3dfb234636bcf63c3340aaab9d P 74077b61d6f54342bf9736b4a9f2eade: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 07:59:09.021718 9790 tablet_bootstrap.cc:492] T 8454f24b0c28451b8ee51be39ca3e07f P a505e7cb133c4392bfda8a4a20874ae9: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:09.022065 9790 tablet_bootstrap.cc:492] T 8454f24b0c28451b8ee51be39ca3e07f P a505e7cb133c4392bfda8a4a20874ae9: Bootstrap complete.
I20240501 07:59:09.022259 9790 ts_tablet_manager.cc:1367] T 8454f24b0c28451b8ee51be39ca3e07f P a505e7cb133c4392bfda8a4a20874ae9: Time spent bootstrapping tablet: real 0.040s user 0.026s sys 0.000s
I20240501 07:59:09.022565 9778 tablet_bootstrap.cc:492] T b7d57b7fd5234f50afbc859bc997f3e1 P 74077b61d6f54342bf9736b4a9f2eade: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:09.022697 9790 raft_consensus.cc:348] T 8454f24b0c28451b8ee51be39ca3e07f P a505e7cb133c4392bfda8a4a20874ae9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: true } }
I20240501 07:59:09.022804 9790 raft_consensus.cc:729] T 8454f24b0c28451b8ee51be39ca3e07f P a505e7cb133c4392bfda8a4a20874ae9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a505e7cb133c4392bfda8a4a20874ae9, State: Initialized, Role: LEARNER
I20240501 07:59:09.022821 9778 tablet_bootstrap.cc:492] T b7d57b7fd5234f50afbc859bc997f3e1 P 74077b61d6f54342bf9736b4a9f2eade: Bootstrap complete.
I20240501 07:59:09.022915 9790 consensus_queue.cc:260] T 8454f24b0c28451b8ee51be39ca3e07f P a505e7cb133c4392bfda8a4a20874ae9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: true } }
I20240501 07:59:09.022964 9778 ts_tablet_manager.cc:1367] T b7d57b7fd5234f50afbc859bc997f3e1 P 74077b61d6f54342bf9736b4a9f2eade: Time spent bootstrapping tablet: real 0.043s user 0.020s sys 0.005s
I20240501 07:59:09.023180 9790 ts_tablet_manager.cc:1398] T 8454f24b0c28451b8ee51be39ca3e07f P a505e7cb133c4392bfda8a4a20874ae9: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:59:09.023324 9778 raft_consensus.cc:348] T b7d57b7fd5234f50afbc859bc997f3e1 P 74077b61d6f54342bf9736b4a9f2eade [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { replace: true } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: true } }
I20240501 07:59:09.023434 9778 raft_consensus.cc:729] T b7d57b7fd5234f50afbc859bc997f3e1 P 74077b61d6f54342bf9736b4a9f2eade [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 74077b61d6f54342bf9736b4a9f2eade, State: Initialized, Role: LEARNER
I20240501 07:59:09.023524 9778 consensus_queue.cc:260] T b7d57b7fd5234f50afbc859bc997f3e1 P 74077b61d6f54342bf9736b4a9f2eade [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { replace: true } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: true } }
I20240501 07:59:09.023666 6728 tablet_copy_service.cc:342] P 1f9cb72be98941b5a8e504cbe34132b9: Request end of tablet copy session a505e7cb133c4392bfda8a4a20874ae9-8454f24b0c28451b8ee51be39ca3e07f received from {username='slave'} at 127.0.99.135:56663
I20240501 07:59:09.023752 6728 tablet_copy_service.cc:433] P 1f9cb72be98941b5a8e504cbe34132b9: ending tablet copy session a505e7cb133c4392bfda8a4a20874ae9-8454f24b0c28451b8ee51be39ca3e07f on tablet 8454f24b0c28451b8ee51be39ca3e07f with peer a505e7cb133c4392bfda8a4a20874ae9
I20240501 07:59:09.011571 9167 tablet_service.cc:1467] Processing CreateTablet for tablet 9a17ba3dfb234636bcf63c3340aaab9d (DEFAULT_TABLE table=rebalancer_dynamic_table [id=81aa8832269b4874bfec689c1e98d696]), partition=
I20240501 07:59:09.023945 9778 ts_tablet_manager.cc:1398] T b7d57b7fd5234f50afbc859bc997f3e1 P 74077b61d6f54342bf9736b4a9f2eade: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240501 07:59:09.024132 9167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a17ba3dfb234636bcf63c3340aaab9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:09.024472 6597 tablet_copy_service.cc:342] P 3b164cbbc00f4e8eb3254ddd349a3ff9: Request end of tablet copy session 74077b61d6f54342bf9736b4a9f2eade-b7d57b7fd5234f50afbc859bc997f3e1 received from {username='slave'} at 127.0.99.133:57559
I20240501 07:59:09.024554 6597 tablet_copy_service.cc:433] P 3b164cbbc00f4e8eb3254ddd349a3ff9: ending tablet copy session 74077b61d6f54342bf9736b4a9f2eade-b7d57b7fd5234f50afbc859bc997f3e1 on tablet b7d57b7fd5234f50afbc859bc997f3e1 with peer 74077b61d6f54342bf9736b4a9f2eade
I20240501 07:59:09.026732 9941 tablet_bootstrap.cc:492] T 9a17ba3dfb234636bcf63c3340aaab9d P 83f20e84af4c40d09a4804b6b26e697f: Bootstrap starting.
I20240501 07:59:09.027525 9854 tablet_bootstrap.cc:492] T 6e2ac8c8857d45b88d3b51147d2609aa P 83f20e84af4c40d09a4804b6b26e697f: Bootstrap complete.
I20240501 07:59:09.028060 9854 ts_tablet_manager.cc:1367] T 6e2ac8c8857d45b88d3b51147d2609aa P 83f20e84af4c40d09a4804b6b26e697f: Time spent bootstrapping tablet: real 0.149s user 0.030s sys 0.000s
I20240501 07:59:09.028167 9941 tablet_bootstrap.cc:654] T 9a17ba3dfb234636bcf63c3340aaab9d P 83f20e84af4c40d09a4804b6b26e697f: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:09.028923 9854 raft_consensus.cc:348] T 6e2ac8c8857d45b88d3b51147d2609aa P 83f20e84af4c40d09a4804b6b26e697f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: true } }
I20240501 07:59:09.029152 9854 raft_consensus.cc:729] T 6e2ac8c8857d45b88d3b51147d2609aa P 83f20e84af4c40d09a4804b6b26e697f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 83f20e84af4c40d09a4804b6b26e697f, State: Initialized, Role: LEARNER
I20240501 07:59:09.029320 9854 consensus_queue.cc:260] T 6e2ac8c8857d45b88d3b51147d2609aa P 83f20e84af4c40d09a4804b6b26e697f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: true } }
I20240501 07:59:09.029673 9854 ts_tablet_manager.cc:1398] T 6e2ac8c8857d45b88d3b51147d2609aa P 83f20e84af4c40d09a4804b6b26e697f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:09.029950 9939 tablet_bootstrap.cc:492] T 9a17ba3dfb234636bcf63c3340aaab9d P 13d05ebdca8c4f489c597ce550dd773d: Bootstrap starting.
I20240501 07:59:09.030864 9939 tablet_bootstrap.cc:654] T 9a17ba3dfb234636bcf63c3340aaab9d P 13d05ebdca8c4f489c597ce550dd773d: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:09.033818 9740 ts_tablet_manager.cc:902] T 7d51b1940b594c79a7d13d7026b76da8 P 83f20e84af4c40d09a4804b6b26e697f: Initiating tablet copy from peer 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132:36773)
I20240501 07:59:09.033816 9489 consensus_queue.cc:1035] T cfcc814f4d2d488ba745ecaf0cc6d61a P 8315d22c56f7439a9e1fc82f153f018c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:09.034298 9740 tablet_copy_client.cc:308] T 7d51b1940b594c79a7d13d7026b76da8 P 83f20e84af4c40d09a4804b6b26e697f: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:36773
I20240501 07:59:09.034696 6728 tablet_copy_service.cc:140] P 1f9cb72be98941b5a8e504cbe34132b9: Received BeginTabletCopySession request for tablet 7d51b1940b594c79a7d13d7026b76da8 from peer 83f20e84af4c40d09a4804b6b26e697f ({username='slave'} at 127.0.99.136:55471)
I20240501 07:59:09.034801 6728 tablet_copy_service.cc:161] P 1f9cb72be98941b5a8e504cbe34132b9: Beginning new tablet copy session on tablet 7d51b1940b594c79a7d13d7026b76da8 from peer 83f20e84af4c40d09a4804b6b26e697f at {username='slave'} at 127.0.99.136:55471: session id = 83f20e84af4c40d09a4804b6b26e697f-7d51b1940b594c79a7d13d7026b76da8
I20240501 07:59:09.035859 6728 tablet_copy_source_session.cc:205] T 7d51b1940b594c79a7d13d7026b76da8 P 1f9cb72be98941b5a8e504cbe34132b9: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:09.036092 9939 tablet_bootstrap.cc:492] T 9a17ba3dfb234636bcf63c3340aaab9d P 13d05ebdca8c4f489c597ce550dd773d: No bootstrap required, opened a new log
I20240501 07:59:09.036243 9939 ts_tablet_manager.cc:1367] T 9a17ba3dfb234636bcf63c3340aaab9d P 13d05ebdca8c4f489c597ce550dd773d: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 07:59:09.036759 9941 tablet_bootstrap.cc:492] T 9a17ba3dfb234636bcf63c3340aaab9d P 83f20e84af4c40d09a4804b6b26e697f: No bootstrap required, opened a new log
I20240501 07:59:09.036696 9939 raft_consensus.cc:348] T 9a17ba3dfb234636bcf63c3340aaab9d P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } }
I20240501 07:59:09.036809 9939 raft_consensus.cc:374] T 9a17ba3dfb234636bcf63c3340aaab9d P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:09.036921 9939 raft_consensus.cc:729] T 9a17ba3dfb234636bcf63c3340aaab9d P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13d05ebdca8c4f489c597ce550dd773d, State: Initialized, Role: FOLLOWER
I20240501 07:59:09.036926 9941 ts_tablet_manager.cc:1367] T 9a17ba3dfb234636bcf63c3340aaab9d P 83f20e84af4c40d09a4804b6b26e697f: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.002s
I20240501 07:59:09.037192 9939 consensus_queue.cc:260] T 9a17ba3dfb234636bcf63c3340aaab9d P 13d05ebdca8c4f489c597ce550dd773d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } }
I20240501 07:59:09.037590 9941 raft_consensus.cc:348] T 9a17ba3dfb234636bcf63c3340aaab9d P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } }
I20240501 07:59:09.037707 9941 raft_consensus.cc:374] T 9a17ba3dfb234636bcf63c3340aaab9d P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:09.037746 9941 raft_consensus.cc:729] T 9a17ba3dfb234636bcf63c3340aaab9d P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83f20e84af4c40d09a4804b6b26e697f, State: Initialized, Role: FOLLOWER
I20240501 07:59:09.037927 9941 consensus_queue.cc:260] T 9a17ba3dfb234636bcf63c3340aaab9d P 83f20e84af4c40d09a4804b6b26e697f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } }
I20240501 07:59:09.038273 9740 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d51b1940b594c79a7d13d7026b76da8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:09.038280 9941 ts_tablet_manager.cc:1398] T 9a17ba3dfb234636bcf63c3340aaab9d P 83f20e84af4c40d09a4804b6b26e697f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:09.038787 9939 ts_tablet_manager.cc:1398] T 9a17ba3dfb234636bcf63c3340aaab9d P 13d05ebdca8c4f489c597ce550dd773d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:59:09.039657 6597 tablet_copy_service.cc:342] P 3b164cbbc00f4e8eb3254ddd349a3ff9: Request end of tablet copy session 83f20e84af4c40d09a4804b6b26e697f-6e2ac8c8857d45b88d3b51147d2609aa received from {username='slave'} at 127.0.99.136:47269
I20240501 07:59:09.039742 6597 tablet_copy_service.cc:433] P 3b164cbbc00f4e8eb3254ddd349a3ff9: ending tablet copy session 83f20e84af4c40d09a4804b6b26e697f-6e2ac8c8857d45b88d3b51147d2609aa on tablet 6e2ac8c8857d45b88d3b51147d2609aa with peer 83f20e84af4c40d09a4804b6b26e697f
I20240501 07:59:09.040489 9740 tablet_copy_client.cc:783] T 7d51b1940b594c79a7d13d7026b76da8 P 83f20e84af4c40d09a4804b6b26e697f: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:09.040635 9740 tablet_copy_client.cc:647] T 7d51b1940b594c79a7d13d7026b76da8 P 83f20e84af4c40d09a4804b6b26e697f: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:09.047188 9740 tablet_copy_client.cc:523] T 7d51b1940b594c79a7d13d7026b76da8 P 83f20e84af4c40d09a4804b6b26e697f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:09.048647 9740 tablet_bootstrap.cc:492] T 7d51b1940b594c79a7d13d7026b76da8 P 83f20e84af4c40d09a4804b6b26e697f: Bootstrap starting.
I20240501 07:59:09.054208 9673 consensus_queue.cc:1035] T c5a876dfb598428ab350593129505504 P 83f20e84af4c40d09a4804b6b26e697f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:09.063110 9258 consensus_queue.cc:1035] T c5a876dfb598428ab350593129505504 P 83f20e84af4c40d09a4804b6b26e697f [LEADER]: Connected to new peer: Peer: permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:09.070924 9116 consensus_queue.cc:1035] T 54b3efabad7f4b608cc9d931546b5743 P 8315d22c56f7439a9e1fc82f153f018c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:09.074968 8512 consensus_queue.cc:1035] T 54b3efabad7f4b608cc9d931546b5743 P 8315d22c56f7439a9e1fc82f153f018c [LEADER]: Connected to new peer: Peer: permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:09.079769 9598 raft_consensus.cc:482] T 9a17ba3dfb234636bcf63c3340aaab9d P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:09.079895 9598 raft_consensus.cc:504] T 9a17ba3dfb234636bcf63c3340aaab9d P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } }
I20240501 07:59:09.080399 9598 leader_election.cc:288] T 9a17ba3dfb234636bcf63c3340aaab9d P 74077b61d6f54342bf9736b4a9f2eade [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134:40141), 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149)
I20240501 07:59:09.081135 9189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a17ba3dfb234636bcf63c3340aaab9d" candidate_uuid: "74077b61d6f54342bf9736b4a9f2eade" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83f20e84af4c40d09a4804b6b26e697f" is_pre_election: true
I20240501 07:59:09.081271 9189 raft_consensus.cc:2454] T 9a17ba3dfb234636bcf63c3340aaab9d P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74077b61d6f54342bf9736b4a9f2eade in term 0.
I20240501 07:59:09.082592 9740 tablet_bootstrap.cc:492] T 7d51b1940b594c79a7d13d7026b76da8 P 83f20e84af4c40d09a4804b6b26e697f: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:09.082814 8728 leader_election.cc:302] T 9a17ba3dfb234636bcf63c3340aaab9d P 74077b61d6f54342bf9736b4a9f2eade [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74077b61d6f54342bf9736b4a9f2eade, 83f20e84af4c40d09a4804b6b26e697f; no voters:
I20240501 07:59:09.084070 9598 raft_consensus.cc:2781] T 9a17ba3dfb234636bcf63c3340aaab9d P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:09.084142 9598 raft_consensus.cc:482] T 9a17ba3dfb234636bcf63c3340aaab9d P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:09.084187 9598 raft_consensus.cc:3037] T 9a17ba3dfb234636bcf63c3340aaab9d P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:09.085338 9598 raft_consensus.cc:504] T 9a17ba3dfb234636bcf63c3340aaab9d P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } }
I20240501 07:59:09.085714 9598 leader_election.cc:288] T 9a17ba3dfb234636bcf63c3340aaab9d P 74077b61d6f54342bf9736b4a9f2eade [CANDIDATE]: Term 1 election: Requested vote from peers 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134:40141), 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149)
I20240501 07:59:09.086472 9188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a17ba3dfb234636bcf63c3340aaab9d" candidate_uuid: "74077b61d6f54342bf9736b4a9f2eade" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83f20e84af4c40d09a4804b6b26e697f"
I20240501 07:59:09.086625 9188 raft_consensus.cc:3037] T 9a17ba3dfb234636bcf63c3340aaab9d P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:09.087231 8924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a17ba3dfb234636bcf63c3340aaab9d" candidate_uuid: "74077b61d6f54342bf9736b4a9f2eade" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13d05ebdca8c4f489c597ce550dd773d" is_pre_election: true
I20240501 07:59:09.087241 8925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a17ba3dfb234636bcf63c3340aaab9d" candidate_uuid: "74077b61d6f54342bf9736b4a9f2eade" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13d05ebdca8c4f489c597ce550dd773d"
I20240501 07:59:09.087395 8924 raft_consensus.cc:2454] T 9a17ba3dfb234636bcf63c3340aaab9d P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74077b61d6f54342bf9736b4a9f2eade in term 0.
I20240501 07:59:09.105187 9740 tablet_bootstrap.cc:492] T 7d51b1940b594c79a7d13d7026b76da8 P 83f20e84af4c40d09a4804b6b26e697f: Bootstrap complete.
I20240501 07:59:09.105443 9740 ts_tablet_manager.cc:1367] T 7d51b1940b594c79a7d13d7026b76da8 P 83f20e84af4c40d09a4804b6b26e697f: Time spent bootstrapping tablet: real 0.057s user 0.027s sys 0.000s
I20240501 07:59:09.105877 9188 raft_consensus.cc:2454] T 9a17ba3dfb234636bcf63c3340aaab9d P 83f20e84af4c40d09a4804b6b26e697f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74077b61d6f54342bf9736b4a9f2eade in term 1.
I20240501 07:59:09.105878 9740 raft_consensus.cc:348] T 7d51b1940b594c79a7d13d7026b76da8 P 83f20e84af4c40d09a4804b6b26e697f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { replace: true } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: true } }
I20240501 07:59:09.106051 9740 raft_consensus.cc:729] T 7d51b1940b594c79a7d13d7026b76da8 P 83f20e84af4c40d09a4804b6b26e697f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 83f20e84af4c40d09a4804b6b26e697f, State: Initialized, Role: LEARNER
I20240501 07:59:09.106158 9740 consensus_queue.cc:260] T 7d51b1940b594c79a7d13d7026b76da8 P 83f20e84af4c40d09a4804b6b26e697f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { replace: true } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: true } }
I20240501 07:59:09.106249 8728 leader_election.cc:302] T 9a17ba3dfb234636bcf63c3340aaab9d P 74077b61d6f54342bf9736b4a9f2eade [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 74077b61d6f54342bf9736b4a9f2eade, 83f20e84af4c40d09a4804b6b26e697f; no voters: 13d05ebdca8c4f489c597ce550dd773d
I20240501 07:59:09.106458 9740 ts_tablet_manager.cc:1398] T 7d51b1940b594c79a7d13d7026b76da8 P 83f20e84af4c40d09a4804b6b26e697f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:09.106696 9598 raft_consensus.cc:2781] T 9a17ba3dfb234636bcf63c3340aaab9d P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:09.106793 9598 raft_consensus.cc:686] T 9a17ba3dfb234636bcf63c3340aaab9d P 74077b61d6f54342bf9736b4a9f2eade [term 1 LEADER]: Becoming Leader. State: Replica: 74077b61d6f54342bf9736b4a9f2eade, State: Running, Role: LEADER
I20240501 07:59:09.106930 9598 consensus_queue.cc:237] T 9a17ba3dfb234636bcf63c3340aaab9d P 74077b61d6f54342bf9736b4a9f2eade [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } }
I20240501 07:59:09.107540 6728 tablet_copy_service.cc:342] P 1f9cb72be98941b5a8e504cbe34132b9: Request end of tablet copy session 83f20e84af4c40d09a4804b6b26e697f-7d51b1940b594c79a7d13d7026b76da8 received from {username='slave'} at 127.0.99.136:55471
I20240501 07:59:09.107630 6728 tablet_copy_service.cc:433] P 1f9cb72be98941b5a8e504cbe34132b9: ending tablet copy session 83f20e84af4c40d09a4804b6b26e697f-7d51b1940b594c79a7d13d7026b76da8 on tablet 7d51b1940b594c79a7d13d7026b76da8 with peer 83f20e84af4c40d09a4804b6b26e697f
I20240501 07:59:09.109102 6168 catalog_manager.cc:5484] T 9a17ba3dfb234636bcf63c3340aaab9d P 74077b61d6f54342bf9736b4a9f2eade reported cstate change: term changed from 0 to 1, leader changed from <none> to 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133). New cstate: current_term: 1 leader_uuid: "74077b61d6f54342bf9736b4a9f2eade" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } health_report { overall_health: HEALTHY } } }
I20240501 07:59:09.112500 9056 raft_consensus.cc:1203] T 26713549654249b09f820a31ee19e2b7 P a505e7cb133c4392bfda8a4a20874ae9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 07:59:09.117607 6172 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55950:
name: "rebalancer_extra_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 07:59:09.124163 9036 tablet_service.cc:1467] Processing CreateTablet for tablet c86c420310ec42ffa5b24304fba74267 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=e37d9a776ed7492ba6f616e4727f982b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:09.124503 9036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c86c420310ec42ffa5b24304fba74267. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:09.124547 8774 tablet_service.cc:1467] Processing CreateTablet for tablet 91a7e5059e2a497fa74b19733c8c6afc (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=e37d9a776ed7492ba6f616e4727f982b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:09.124718 9167 tablet_service.cc:1467] Processing CreateTablet for tablet c86c420310ec42ffa5b24304fba74267 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=e37d9a776ed7492ba6f616e4727f982b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:09.124857 8774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91a7e5059e2a497fa74b19733c8c6afc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:09.125015 9167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c86c420310ec42ffa5b24304fba74267. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:09.125476 6424 tablet_service.cc:1467] Processing CreateTablet for tablet c86c420310ec42ffa5b24304fba74267 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=e37d9a776ed7492ba6f616e4727f982b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:09.125723 6424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c86c420310ec42ffa5b24304fba74267. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:09.127172 8757 tablet_service.cc:1467] Processing CreateTablet for tablet 28eaf22342ff4b5394f958d0b659273c (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=e37d9a776ed7492ba6f616e4727f982b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:09.127250 6686 tablet_service.cc:1467] Processing CreateTablet for tablet 28eaf22342ff4b5394f958d0b659273c (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=e37d9a776ed7492ba6f616e4727f982b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:09.127382 8757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28eaf22342ff4b5394f958d0b659273c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:09.127483 6686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28eaf22342ff4b5394f958d0b659273c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:09.127950 10002 tablet_bootstrap.cc:492] T c86c420310ec42ffa5b24304fba74267 P 8315d22c56f7439a9e1fc82f153f018c: Bootstrap starting.
I20240501 07:59:09.128145 6294 tablet_service.cc:1467] Processing CreateTablet for tablet 91a7e5059e2a497fa74b19733c8c6afc (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=e37d9a776ed7492ba6f616e4727f982b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:09.128427 6294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91a7e5059e2a497fa74b19733c8c6afc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:09.129067 10002 tablet_bootstrap.cc:654] T c86c420310ec42ffa5b24304fba74267 P 8315d22c56f7439a9e1fc82f153f018c: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:09.129521 9392 tablet_bootstrap.cc:492] T 91a7e5059e2a497fa74b19733c8c6afc P 74077b61d6f54342bf9736b4a9f2eade: Bootstrap starting.
I20240501 07:59:09.130370 10004 tablet_bootstrap.cc:492] T 28eaf22342ff4b5394f958d0b659273c P 1f9cb72be98941b5a8e504cbe34132b9: Bootstrap starting.
I20240501 07:59:09.130618 9392 tablet_bootstrap.cc:654] T 91a7e5059e2a497fa74b19733c8c6afc P 74077b61d6f54342bf9736b4a9f2eade: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:09.130571 8905 tablet_service.cc:1467] Processing CreateTablet for tablet 91a7e5059e2a497fa74b19733c8c6afc (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=e37d9a776ed7492ba6f616e4727f982b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:09.130841 8905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91a7e5059e2a497fa74b19733c8c6afc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:09.131182 10004 tablet_bootstrap.cc:654] T 28eaf22342ff4b5394f958d0b659273c P 1f9cb72be98941b5a8e504cbe34132b9: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:09.131227 9941 tablet_bootstrap.cc:492] T c86c420310ec42ffa5b24304fba74267 P 83f20e84af4c40d09a4804b6b26e697f: Bootstrap starting.
I20240501 07:59:09.132200 9941 tablet_bootstrap.cc:654] T c86c420310ec42ffa5b24304fba74267 P 83f20e84af4c40d09a4804b6b26e697f: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:09.132742 9939 tablet_bootstrap.cc:492] T 91a7e5059e2a497fa74b19733c8c6afc P 13d05ebdca8c4f489c597ce550dd773d: Bootstrap starting.
I20240501 07:59:09.133244 10006 tablet_bootstrap.cc:492] T 91a7e5059e2a497fa74b19733c8c6afc P 8b8b3ad3aa8f4053b2167dd550f2086f: Bootstrap starting.
I20240501 07:59:09.134029 9939 tablet_bootstrap.cc:654] T 91a7e5059e2a497fa74b19733c8c6afc P 13d05ebdca8c4f489c597ce550dd773d: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:09.134384 9392 tablet_bootstrap.cc:492] T 91a7e5059e2a497fa74b19733c8c6afc P 74077b61d6f54342bf9736b4a9f2eade: No bootstrap required, opened a new log
I20240501 07:59:09.134390 9036 tablet_service.cc:1467] Processing CreateTablet for tablet 28eaf22342ff4b5394f958d0b659273c (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=e37d9a776ed7492ba6f616e4727f982b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:09.134465 9392 ts_tablet_manager.cc:1367] T 91a7e5059e2a497fa74b19733c8c6afc P 74077b61d6f54342bf9736b4a9f2eade: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.001s
I20240501 07:59:09.134665 9036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28eaf22342ff4b5394f958d0b659273c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:09.134997 10006 tablet_bootstrap.cc:654] T 91a7e5059e2a497fa74b19733c8c6afc P 8b8b3ad3aa8f4053b2167dd550f2086f: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:09.134979 9392 raft_consensus.cc:348] T 91a7e5059e2a497fa74b19733c8c6afc P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } }
I20240501 07:59:09.135092 9392 raft_consensus.cc:374] T 91a7e5059e2a497fa74b19733c8c6afc P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:09.135125 9392 raft_consensus.cc:729] T 91a7e5059e2a497fa74b19733c8c6afc P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74077b61d6f54342bf9736b4a9f2eade, State: Initialized, Role: FOLLOWER
I20240501 07:59:09.135242 9392 consensus_queue.cc:260] T 91a7e5059e2a497fa74b19733c8c6afc P 74077b61d6f54342bf9736b4a9f2eade [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } }
I20240501 07:59:09.135661 9392 ts_tablet_manager.cc:1398] T 91a7e5059e2a497fa74b19733c8c6afc P 74077b61d6f54342bf9736b4a9f2eade: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:09.136503 9392 tablet_bootstrap.cc:492] T 28eaf22342ff4b5394f958d0b659273c P 74077b61d6f54342bf9736b4a9f2eade: Bootstrap starting.
I20240501 07:59:09.137300 9939 tablet_bootstrap.cc:492] T 91a7e5059e2a497fa74b19733c8c6afc P 13d05ebdca8c4f489c597ce550dd773d: No bootstrap required, opened a new log
I20240501 07:59:09.137392 9939 ts_tablet_manager.cc:1367] T 91a7e5059e2a497fa74b19733c8c6afc P 13d05ebdca8c4f489c597ce550dd773d: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 07:59:09.137848 9939 raft_consensus.cc:348] T 91a7e5059e2a497fa74b19733c8c6afc P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } }
I20240501 07:59:09.137974 9939 raft_consensus.cc:374] T 91a7e5059e2a497fa74b19733c8c6afc P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:09.138010 9939 raft_consensus.cc:729] T 91a7e5059e2a497fa74b19733c8c6afc P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13d05ebdca8c4f489c597ce550dd773d, State: Initialized, Role: FOLLOWER
I20240501 07:59:09.138145 9939 consensus_queue.cc:260] T 91a7e5059e2a497fa74b19733c8c6afc P 13d05ebdca8c4f489c597ce550dd773d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } }
I20240501 07:59:09.138434 9392 tablet_bootstrap.cc:654] T 28eaf22342ff4b5394f958d0b659273c P 74077b61d6f54342bf9736b4a9f2eade: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:09.139843 10001 tablet_bootstrap.cc:492] T c86c420310ec42ffa5b24304fba74267 P a505e7cb133c4392bfda8a4a20874ae9: Bootstrap starting.
I20240501 07:59:09.140071 9941 tablet_bootstrap.cc:492] T c86c420310ec42ffa5b24304fba74267 P 83f20e84af4c40d09a4804b6b26e697f: No bootstrap required, opened a new log
I20240501 07:59:09.140164 9941 ts_tablet_manager.cc:1367] T c86c420310ec42ffa5b24304fba74267 P 83f20e84af4c40d09a4804b6b26e697f: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.001s
I20240501 07:59:09.140686 9941 raft_consensus.cc:348] T c86c420310ec42ffa5b24304fba74267 P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } }
I20240501 07:59:09.140803 9941 raft_consensus.cc:374] T c86c420310ec42ffa5b24304fba74267 P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:09.140844 9941 raft_consensus.cc:729] T c86c420310ec42ffa5b24304fba74267 P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83f20e84af4c40d09a4804b6b26e697f, State: Initialized, Role: FOLLOWER
I20240501 07:59:09.140882 10001 tablet_bootstrap.cc:654] T c86c420310ec42ffa5b24304fba74267 P a505e7cb133c4392bfda8a4a20874ae9: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:09.141016 9941 consensus_queue.cc:260] T c86c420310ec42ffa5b24304fba74267 P 83f20e84af4c40d09a4804b6b26e697f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } }
I20240501 07:59:09.141273 9939 ts_tablet_manager.cc:1398] T 91a7e5059e2a497fa74b19733c8c6afc P 13d05ebdca8c4f489c597ce550dd773d: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240501 07:59:09.141319 9941 ts_tablet_manager.cc:1398] T c86c420310ec42ffa5b24304fba74267 P 83f20e84af4c40d09a4804b6b26e697f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:09.142452 10004 tablet_bootstrap.cc:492] T 28eaf22342ff4b5394f958d0b659273c P 1f9cb72be98941b5a8e504cbe34132b9: No bootstrap required, opened a new log
I20240501 07:59:09.142508 10002 tablet_bootstrap.cc:492] T c86c420310ec42ffa5b24304fba74267 P 8315d22c56f7439a9e1fc82f153f018c: No bootstrap required, opened a new log
I20240501 07:59:09.142555 10004 ts_tablet_manager.cc:1367] T 28eaf22342ff4b5394f958d0b659273c P 1f9cb72be98941b5a8e504cbe34132b9: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.000s
I20240501 07:59:09.142599 10002 ts_tablet_manager.cc:1367] T c86c420310ec42ffa5b24304fba74267 P 8315d22c56f7439a9e1fc82f153f018c: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.001s
I20240501 07:59:09.142647 10001 tablet_bootstrap.cc:492] T c86c420310ec42ffa5b24304fba74267 P a505e7cb133c4392bfda8a4a20874ae9: No bootstrap required, opened a new log
I20240501 07:59:09.142742 10001 ts_tablet_manager.cc:1367] T c86c420310ec42ffa5b24304fba74267 P a505e7cb133c4392bfda8a4a20874ae9: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20240501 07:59:09.143024 10002 raft_consensus.cc:348] T c86c420310ec42ffa5b24304fba74267 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } }
I20240501 07:59:09.143018 10004 raft_consensus.cc:348] T 28eaf22342ff4b5394f958d0b659273c P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:59:09.143132 10004 raft_consensus.cc:374] T 28eaf22342ff4b5394f958d0b659273c P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:09.143132 10002 raft_consensus.cc:374] T c86c420310ec42ffa5b24304fba74267 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:09.143167 10002 raft_consensus.cc:729] T c86c420310ec42ffa5b24304fba74267 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8315d22c56f7439a9e1fc82f153f018c, State: Initialized, Role: FOLLOWER
I20240501 07:59:09.143167 10004 raft_consensus.cc:729] T 28eaf22342ff4b5394f958d0b659273c P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f9cb72be98941b5a8e504cbe34132b9, State: Initialized, Role: FOLLOWER
I20240501 07:59:09.143286 10002 consensus_queue.cc:260] T c86c420310ec42ffa5b24304fba74267 P 8315d22c56f7439a9e1fc82f153f018c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } }
I20240501 07:59:09.143287 10004 consensus_queue.cc:260] T 28eaf22342ff4b5394f958d0b659273c P 1f9cb72be98941b5a8e504cbe34132b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:59:09.143576 10004 ts_tablet_manager.cc:1398] T 28eaf22342ff4b5394f958d0b659273c P 1f9cb72be98941b5a8e504cbe34132b9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:09.143599 10002 ts_tablet_manager.cc:1398] T c86c420310ec42ffa5b24304fba74267 P 8315d22c56f7439a9e1fc82f153f018c: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 07:59:09.143906 10001 raft_consensus.cc:348] T c86c420310ec42ffa5b24304fba74267 P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } }
I20240501 07:59:09.144014 10001 raft_consensus.cc:374] T c86c420310ec42ffa5b24304fba74267 P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:09.144043 10001 raft_consensus.cc:729] T c86c420310ec42ffa5b24304fba74267 P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a505e7cb133c4392bfda8a4a20874ae9, State: Initialized, Role: FOLLOWER
I20240501 07:59:09.144197 10001 consensus_queue.cc:260] T c86c420310ec42ffa5b24304fba74267 P a505e7cb133c4392bfda8a4a20874ae9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } }
I20240501 07:59:09.144464 10001 ts_tablet_manager.cc:1398] T c86c420310ec42ffa5b24304fba74267 P a505e7cb133c4392bfda8a4a20874ae9: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 07:59:09.144608 10001 tablet_bootstrap.cc:492] T 28eaf22342ff4b5394f958d0b659273c P a505e7cb133c4392bfda8a4a20874ae9: Bootstrap starting.
I20240501 07:59:09.145558 10001 tablet_bootstrap.cc:654] T 28eaf22342ff4b5394f958d0b659273c P a505e7cb133c4392bfda8a4a20874ae9: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:09.146832 10006 tablet_bootstrap.cc:492] T 91a7e5059e2a497fa74b19733c8c6afc P 8b8b3ad3aa8f4053b2167dd550f2086f: No bootstrap required, opened a new log
I20240501 07:59:09.146945 10006 ts_tablet_manager.cc:1367] T 91a7e5059e2a497fa74b19733c8c6afc P 8b8b3ad3aa8f4053b2167dd550f2086f: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.000s
I20240501 07:59:09.147424 10006 raft_consensus.cc:348] T 91a7e5059e2a497fa74b19733c8c6afc P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } }
I20240501 07:59:09.148156 10006 raft_consensus.cc:374] T 91a7e5059e2a497fa74b19733c8c6afc P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:09.148849 10001 tablet_bootstrap.cc:492] T 28eaf22342ff4b5394f958d0b659273c P a505e7cb133c4392bfda8a4a20874ae9: No bootstrap required, opened a new log
I20240501 07:59:09.149024 10001 ts_tablet_manager.cc:1367] T 28eaf22342ff4b5394f958d0b659273c P a505e7cb133c4392bfda8a4a20874ae9: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 07:59:09.149529 10001 raft_consensus.cc:348] T 28eaf22342ff4b5394f958d0b659273c P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:59:09.149642 10001 raft_consensus.cc:374] T 28eaf22342ff4b5394f958d0b659273c P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:09.149672 10001 raft_consensus.cc:729] T 28eaf22342ff4b5394f958d0b659273c P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a505e7cb133c4392bfda8a4a20874ae9, State: Initialized, Role: FOLLOWER
I20240501 07:59:09.149787 10001 consensus_queue.cc:260] T 28eaf22342ff4b5394f958d0b659273c P a505e7cb133c4392bfda8a4a20874ae9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:59:09.150022 10001 ts_tablet_manager.cc:1398] T 28eaf22342ff4b5394f958d0b659273c P a505e7cb133c4392bfda8a4a20874ae9: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:59:09.150154 10006 raft_consensus.cc:729] T 91a7e5059e2a497fa74b19733c8c6afc P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b8b3ad3aa8f4053b2167dd550f2086f, State: Initialized, Role: FOLLOWER
I20240501 07:59:09.150352 10006 consensus_queue.cc:260] T 91a7e5059e2a497fa74b19733c8c6afc P 8b8b3ad3aa8f4053b2167dd550f2086f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } }
I20240501 07:59:09.150687 10006 ts_tablet_manager.cc:1398] T 91a7e5059e2a497fa74b19733c8c6afc P 8b8b3ad3aa8f4053b2167dd550f2086f: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240501 07:59:09.157392 9392 tablet_bootstrap.cc:492] T 28eaf22342ff4b5394f958d0b659273c P 74077b61d6f54342bf9736b4a9f2eade: No bootstrap required, opened a new log
I20240501 07:59:09.157521 9392 ts_tablet_manager.cc:1367] T 28eaf22342ff4b5394f958d0b659273c P 74077b61d6f54342bf9736b4a9f2eade: Time spent bootstrapping tablet: real 0.021s user 0.002s sys 0.001s
I20240501 07:59:09.158071 9392 raft_consensus.cc:348] T 28eaf22342ff4b5394f958d0b659273c P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:59:09.158190 9392 raft_consensus.cc:374] T 28eaf22342ff4b5394f958d0b659273c P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:09.158222 9392 raft_consensus.cc:729] T 28eaf22342ff4b5394f958d0b659273c P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74077b61d6f54342bf9736b4a9f2eade, State: Initialized, Role: FOLLOWER
I20240501 07:59:09.158340 9392 consensus_queue.cc:260] T 28eaf22342ff4b5394f958d0b659273c P 74077b61d6f54342bf9736b4a9f2eade [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:59:09.158664 9392 ts_tablet_manager.cc:1398] T 28eaf22342ff4b5394f958d0b659273c P 74077b61d6f54342bf9736b4a9f2eade: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:59:09.192937 8924 raft_consensus.cc:1203] T ec2d3ccf0937430f947f3e2bde0506c5 P 13d05ebdca8c4f489c597ce550dd773d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 07:59:09.194664 9673 raft_consensus.cc:482] T c86c420310ec42ffa5b24304fba74267 P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:09.194787 9673 raft_consensus.cc:504] T c86c420310ec42ffa5b24304fba74267 P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } }
I20240501 07:59:09.195315 9673 leader_election.cc:288] T c86c420310ec42ffa5b24304fba74267 P 83f20e84af4c40d09a4804b6b26e697f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607), 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130:32941)
I20240501 07:59:09.195698 6445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c86c420310ec42ffa5b24304fba74267" candidate_uuid: "83f20e84af4c40d09a4804b6b26e697f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8315d22c56f7439a9e1fc82f153f018c" is_pre_election: true
I20240501 07:59:09.195842 6445 raft_consensus.cc:2454] T c86c420310ec42ffa5b24304fba74267 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83f20e84af4c40d09a4804b6b26e697f in term 0.
I20240501 07:59:09.196204 9124 leader_election.cc:302] T c86c420310ec42ffa5b24304fba74267 P 83f20e84af4c40d09a4804b6b26e697f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8315d22c56f7439a9e1fc82f153f018c, 83f20e84af4c40d09a4804b6b26e697f; no voters:
I20240501 07:59:09.196431 9673 raft_consensus.cc:2781] T c86c420310ec42ffa5b24304fba74267 P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:09.196492 9673 raft_consensus.cc:482] T c86c420310ec42ffa5b24304fba74267 P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:09.196543 9673 raft_consensus.cc:3037] T c86c420310ec42ffa5b24304fba74267 P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:09.197674 9673 raft_consensus.cc:504] T c86c420310ec42ffa5b24304fba74267 P 83f20e84af4c40d09a4804b6b26e697f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } }
I20240501 07:59:09.198015 9673 leader_election.cc:288] T c86c420310ec42ffa5b24304fba74267 P 83f20e84af4c40d09a4804b6b26e697f [CANDIDATE]: Term 1 election: Requested vote from peers a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607), 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130:32941)
I20240501 07:59:09.199909 6445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c86c420310ec42ffa5b24304fba74267" candidate_uuid: "83f20e84af4c40d09a4804b6b26e697f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8315d22c56f7439a9e1fc82f153f018c"
I20240501 07:59:09.200039 6445 raft_consensus.cc:3037] T c86c420310ec42ffa5b24304fba74267 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:09.200341 9188 raft_consensus.cc:1203] T d3f2250a62c6429e886753aaffcbe883 P 83f20e84af4c40d09a4804b6b26e697f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 07:59:09.201205 6445 raft_consensus.cc:2454] T c86c420310ec42ffa5b24304fba74267 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83f20e84af4c40d09a4804b6b26e697f in term 1.
I20240501 07:59:09.201515 9124 leader_election.cc:302] T c86c420310ec42ffa5b24304fba74267 P 83f20e84af4c40d09a4804b6b26e697f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8315d22c56f7439a9e1fc82f153f018c, 83f20e84af4c40d09a4804b6b26e697f; no voters:
I20240501 07:59:09.201809 9056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c86c420310ec42ffa5b24304fba74267" candidate_uuid: "83f20e84af4c40d09a4804b6b26e697f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a505e7cb133c4392bfda8a4a20874ae9" is_pre_election: true
I20240501 07:59:09.201972 9056 raft_consensus.cc:2454] T c86c420310ec42ffa5b24304fba74267 P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83f20e84af4c40d09a4804b6b26e697f in term 0.
I20240501 07:59:09.202093 9673 raft_consensus.cc:2781] T c86c420310ec42ffa5b24304fba74267 P 83f20e84af4c40d09a4804b6b26e697f [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:09.202136 9055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c86c420310ec42ffa5b24304fba74267" candidate_uuid: "83f20e84af4c40d09a4804b6b26e697f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a505e7cb133c4392bfda8a4a20874ae9"
I20240501 07:59:09.202178 9673 raft_consensus.cc:686] T c86c420310ec42ffa5b24304fba74267 P 83f20e84af4c40d09a4804b6b26e697f [term 1 LEADER]: Becoming Leader. State: Replica: 83f20e84af4c40d09a4804b6b26e697f, State: Running, Role: LEADER
I20240501 07:59:09.202240 9055 raft_consensus.cc:3037] T c86c420310ec42ffa5b24304fba74267 P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:09.202308 9673 consensus_queue.cc:237] T c86c420310ec42ffa5b24304fba74267 P 83f20e84af4c40d09a4804b6b26e697f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } }
I20240501 07:59:09.203269 9055 raft_consensus.cc:2454] T c86c420310ec42ffa5b24304fba74267 P a505e7cb133c4392bfda8a4a20874ae9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83f20e84af4c40d09a4804b6b26e697f in term 1.
I20240501 07:59:09.205050 6172 catalog_manager.cc:5484] T c86c420310ec42ffa5b24304fba74267 P 83f20e84af4c40d09a4804b6b26e697f reported cstate change: term changed from 0 to 1, leader changed from <none> to 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136). New cstate: current_term: 1 leader_uuid: "83f20e84af4c40d09a4804b6b26e697f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } health_report { overall_health: UNKNOWN } } }
I20240501 07:59:09.222004 6167 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55950:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 07:59:09.227288 8924 raft_consensus.cc:3037] T 9a17ba3dfb234636bcf63c3340aaab9d P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:09.227355 9188 raft_consensus.cc:1261] T 9a17ba3dfb234636bcf63c3340aaab9d P 83f20e84af4c40d09a4804b6b26e697f [term 1 FOLLOWER]: Refusing update from remote peer 74077b61d6f54342bf9736b4a9f2eade: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:09.228494 8924 raft_consensus.cc:1261] T 9a17ba3dfb234636bcf63c3340aaab9d P 13d05ebdca8c4f489c597ce550dd773d [term 1 FOLLOWER]: Refusing update from remote peer 74077b61d6f54342bf9736b4a9f2eade: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:09.228715 9406 consensus_queue.cc:1035] T 9a17ba3dfb234636bcf63c3340aaab9d P 74077b61d6f54342bf9736b4a9f2eade [LEADER]: Connected to new peer: Peer: permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:09.229066 9598 consensus_queue.cc:1035] T 9a17ba3dfb234636bcf63c3340aaab9d P 74077b61d6f54342bf9736b4a9f2eade [LEADER]: Connected to new peer: Peer: permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:09.229133 10040 raft_consensus.cc:482] T 91a7e5059e2a497fa74b19733c8c6afc P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:09.229236 10040 raft_consensus.cc:504] T 91a7e5059e2a497fa74b19733c8c6afc P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } }
I20240501 07:59:09.229640 10040 leader_election.cc:288] T 91a7e5059e2a497fa74b19733c8c6afc P 8b8b3ad3aa8f4053b2167dd550f2086f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134:40141), 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133:39859)
I20240501 07:59:09.230005 8794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a7e5059e2a497fa74b19733c8c6afc" candidate_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74077b61d6f54342bf9736b4a9f2eade" is_pre_election: true
I20240501 07:59:09.230572 8925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a7e5059e2a497fa74b19733c8c6afc" candidate_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13d05ebdca8c4f489c597ce550dd773d" is_pre_election: true
I20240501 07:59:09.230695 8925 raft_consensus.cc:2454] T 91a7e5059e2a497fa74b19733c8c6afc P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b8b3ad3aa8f4053b2167dd550f2086f in term 0.
I20240501 07:59:09.232137 6250 leader_election.cc:302] T 91a7e5059e2a497fa74b19733c8c6afc P 8b8b3ad3aa8f4053b2167dd550f2086f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13d05ebdca8c4f489c597ce550dd773d, 8b8b3ad3aa8f4053b2167dd550f2086f; no voters:
I20240501 07:59:09.232345 9675 tablet.cc:1722] T 9a17ba3dfb234636bcf63c3340aaab9d P 74077b61d6f54342bf9736b4a9f2eade: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 07:59:09.232412 10040 raft_consensus.cc:2781] T 91a7e5059e2a497fa74b19733c8c6afc P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:09.232481 10040 raft_consensus.cc:482] T 91a7e5059e2a497fa74b19733c8c6afc P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:09.229384 10069 raft_consensus.cc:482] T 91a7e5059e2a497fa74b19733c8c6afc P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:09.232522 10040 raft_consensus.cc:3037] T 91a7e5059e2a497fa74b19733c8c6afc P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:09.232573 10069 raft_consensus.cc:504] T 91a7e5059e2a497fa74b19733c8c6afc P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } }
I20240501 07:59:09.232669 9675 tablet.cc:1620] T 9a17ba3dfb234636bcf63c3340aaab9d P 74077b61d6f54342bf9736b4a9f2eade: MemRowSet was empty: no flush needed.
I20240501 07:59:09.232744 8794 raft_consensus.cc:2454] T 91a7e5059e2a497fa74b19733c8c6afc P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b8b3ad3aa8f4053b2167dd550f2086f in term 0.
I20240501 07:59:09.233289 9582 mvcc.cc:205] Tried to move back new op lower bound from 7022798230430691328 to 7022798229943341056. Current Snapshot: MvccSnapshot[applied={T|T < 7022798230430691328}]
I20240501 07:59:09.233690 10040 raft_consensus.cc:504] T 91a7e5059e2a497fa74b19733c8c6afc P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } }
I20240501 07:59:09.233831 10069 leader_election.cc:288] T 91a7e5059e2a497fa74b19733c8c6afc P 74077b61d6f54342bf9736b4a9f2eade [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129:46875), 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134:40141)
I20240501 07:59:09.234048 10040 leader_election.cc:288] T 91a7e5059e2a497fa74b19733c8c6afc P 8b8b3ad3aa8f4053b2167dd550f2086f [CANDIDATE]: Term 1 election: Requested vote from peers 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134:40141), 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133:39859)
I20240501 07:59:09.234593 8794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a7e5059e2a497fa74b19733c8c6afc" candidate_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74077b61d6f54342bf9736b4a9f2eade"
I20240501 07:59:09.234702 8794 raft_consensus.cc:3037] T 91a7e5059e2a497fa74b19733c8c6afc P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:09.235777 8794 raft_consensus.cc:2454] T 91a7e5059e2a497fa74b19733c8c6afc P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b8b3ad3aa8f4053b2167dd550f2086f in term 1.
I20240501 07:59:09.236326 9672 tablet.cc:1722] T 9a17ba3dfb234636bcf63c3340aaab9d P 83f20e84af4c40d09a4804b6b26e697f: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 07:59:09.236658 9672 tablet.cc:1620] T 9a17ba3dfb234636bcf63c3340aaab9d P 83f20e84af4c40d09a4804b6b26e697f: MemRowSet was empty: no flush needed.
I20240501 07:59:09.237181 8925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a7e5059e2a497fa74b19733c8c6afc" candidate_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13d05ebdca8c4f489c597ce550dd773d"
I20240501 07:59:09.237291 8925 raft_consensus.cc:3037] T 91a7e5059e2a497fa74b19733c8c6afc P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:09.238027 6168 catalog_manager.cc:5576] TS 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134:40141) does not have the latest schema for tablet 9a17ba3dfb234636bcf63c3340aaab9d (table rebalancer_dynamic_table [id=81aa8832269b4874bfec689c1e98d696]). Expected version 1 got 0
I20240501 07:59:09.238363 8925 raft_consensus.cc:2454] T 91a7e5059e2a497fa74b19733c8c6afc P 13d05ebdca8c4f489c597ce550dd773d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b8b3ad3aa8f4053b2167dd550f2086f in term 1.
I20240501 07:59:09.238590 6172 catalog_manager.cc:5576] TS 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149) does not have the latest schema for tablet 9a17ba3dfb234636bcf63c3340aaab9d (table rebalancer_dynamic_table [id=81aa8832269b4874bfec689c1e98d696]). Expected version 1 got 0
I20240501 07:59:09.238862 9940 raft_consensus.cc:482] T 28eaf22342ff4b5394f958d0b659273c P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:09.238956 9940 raft_consensus.cc:504] T 28eaf22342ff4b5394f958d0b659273c P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:59:09.239490 9940 leader_election.cc:288] T 28eaf22342ff4b5394f958d0b659273c P a505e7cb133c4392bfda8a4a20874ae9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133:39859), 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132:36773)
I20240501 07:59:09.241328 6708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28eaf22342ff4b5394f958d0b659273c" candidate_uuid: "a505e7cb133c4392bfda8a4a20874ae9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f9cb72be98941b5a8e504cbe34132b9" is_pre_election: true
I20240501 07:59:09.241480 6708 raft_consensus.cc:2454] T 28eaf22342ff4b5394f958d0b659273c P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a505e7cb133c4392bfda8a4a20874ae9 in term 0.
I20240501 07:59:09.243002 8991 leader_election.cc:302] T 28eaf22342ff4b5394f958d0b659273c P a505e7cb133c4392bfda8a4a20874ae9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f9cb72be98941b5a8e504cbe34132b9, a505e7cb133c4392bfda8a4a20874ae9; no voters:
I20240501 07:59:09.244329 6168 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 0bc706e85ce74cef97a4827ea4d4aa65: rebalancer_dynamic_table [id=81aa8832269b4874bfec689c1e98d696] alter complete (version 1)
I20240501 07:59:09.245106 6250 leader_election.cc:302] T 91a7e5059e2a497fa74b19733c8c6afc P 8b8b3ad3aa8f4053b2167dd550f2086f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13d05ebdca8c4f489c597ce550dd773d, 8b8b3ad3aa8f4053b2167dd550f2086f; no voters:
I20240501 07:59:09.245952 6314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a7e5059e2a497fa74b19733c8c6afc" candidate_uuid: "74077b61d6f54342bf9736b4a9f2eade" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" is_pre_election: true
I20240501 07:59:09.246129 6314 raft_consensus.cc:2379] T 91a7e5059e2a497fa74b19733c8c6afc P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 74077b61d6f54342bf9736b4a9f2eade in current term 1: Already voted for candidate 8b8b3ad3aa8f4053b2167dd550f2086f in this term.
I20240501 07:59:09.247848 8794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28eaf22342ff4b5394f958d0b659273c" candidate_uuid: "a505e7cb133c4392bfda8a4a20874ae9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74077b61d6f54342bf9736b4a9f2eade" is_pre_election: true
I20240501 07:59:09.247903 9940 raft_consensus.cc:2781] T 28eaf22342ff4b5394f958d0b659273c P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:09.247978 9940 raft_consensus.cc:482] T 28eaf22342ff4b5394f958d0b659273c P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:09.248021 9940 raft_consensus.cc:3037] T 28eaf22342ff4b5394f958d0b659273c P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:09.248016 8794 raft_consensus.cc:2454] T 28eaf22342ff4b5394f958d0b659273c P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a505e7cb133c4392bfda8a4a20874ae9 in term 0.
I20240501 07:59:09.248932 9940 raft_consensus.cc:504] T 28eaf22342ff4b5394f958d0b659273c P a505e7cb133c4392bfda8a4a20874ae9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:59:09.249307 9940 leader_election.cc:288] T 28eaf22342ff4b5394f958d0b659273c P a505e7cb133c4392bfda8a4a20874ae9 [CANDIDATE]: Term 1 election: Requested vote from peers 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133:39859), 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132:36773)
I20240501 07:59:09.250144 8794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28eaf22342ff4b5394f958d0b659273c" candidate_uuid: "a505e7cb133c4392bfda8a4a20874ae9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74077b61d6f54342bf9736b4a9f2eade"
I20240501 07:59:09.250154 6708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28eaf22342ff4b5394f958d0b659273c" candidate_uuid: "a505e7cb133c4392bfda8a4a20874ae9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f9cb72be98941b5a8e504cbe34132b9"
I20240501 07:59:09.250257 8794 raft_consensus.cc:3037] T 28eaf22342ff4b5394f958d0b659273c P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:09.250272 6708 raft_consensus.cc:3037] T 28eaf22342ff4b5394f958d0b659273c P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:09.251277 6708 raft_consensus.cc:2454] T 28eaf22342ff4b5394f958d0b659273c P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a505e7cb133c4392bfda8a4a20874ae9 in term 1.
I20240501 07:59:09.251286 8794 raft_consensus.cc:2454] T 28eaf22342ff4b5394f958d0b659273c P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a505e7cb133c4392bfda8a4a20874ae9 in term 1.
I20240501 07:59:09.251574 8991 leader_election.cc:302] T 28eaf22342ff4b5394f958d0b659273c P a505e7cb133c4392bfda8a4a20874ae9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f9cb72be98941b5a8e504cbe34132b9, a505e7cb133c4392bfda8a4a20874ae9; no voters:
I20240501 07:59:09.251791 9940 raft_consensus.cc:2781] T 28eaf22342ff4b5394f958d0b659273c P a505e7cb133c4392bfda8a4a20874ae9 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:09.251858 9940 raft_consensus.cc:686] T 28eaf22342ff4b5394f958d0b659273c P a505e7cb133c4392bfda8a4a20874ae9 [term 1 LEADER]: Becoming Leader. State: Replica: a505e7cb133c4392bfda8a4a20874ae9, State: Running, Role: LEADER
I20240501 07:59:09.251979 9940 consensus_queue.cc:237] T 28eaf22342ff4b5394f958d0b659273c P a505e7cb133c4392bfda8a4a20874ae9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:59:09.253099 8925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a7e5059e2a497fa74b19733c8c6afc" candidate_uuid: "74077b61d6f54342bf9736b4a9f2eade" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13d05ebdca8c4f489c597ce550dd773d" is_pre_election: true
I20240501 07:59:09.253280 8925 raft_consensus.cc:2379] T 91a7e5059e2a497fa74b19733c8c6afc P 13d05ebdca8c4f489c597ce550dd773d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 74077b61d6f54342bf9736b4a9f2eade in current term 1: Already voted for candidate 8b8b3ad3aa8f4053b2167dd550f2086f in this term.
I20240501 07:59:09.253509 10040 raft_consensus.cc:2781] T 91a7e5059e2a497fa74b19733c8c6afc P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:09.253623 10040 raft_consensus.cc:686] T 91a7e5059e2a497fa74b19733c8c6afc P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 LEADER]: Becoming Leader. State: Replica: 8b8b3ad3aa8f4053b2167dd550f2086f, State: Running, Role: LEADER
I20240501 07:59:09.253759 10040 consensus_queue.cc:237] T 91a7e5059e2a497fa74b19733c8c6afc P 8b8b3ad3aa8f4053b2167dd550f2086f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } }
I20240501 07:59:09.254570 8729 leader_election.cc:302] T 91a7e5059e2a497fa74b19733c8c6afc P 74077b61d6f54342bf9736b4a9f2eade [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74077b61d6f54342bf9736b4a9f2eade; no voters: 13d05ebdca8c4f489c597ce550dd773d, 8b8b3ad3aa8f4053b2167dd550f2086f
I20240501 07:59:09.254902 9598 raft_consensus.cc:2726] T 91a7e5059e2a497fa74b19733c8c6afc P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 07:59:09.255648 6165 catalog_manager.cc:5484] T 28eaf22342ff4b5394f958d0b659273c P a505e7cb133c4392bfda8a4a20874ae9 reported cstate change: term changed from 0 to 1, leader changed from <none> to a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135). New cstate: current_term: 1 leader_uuid: "a505e7cb133c4392bfda8a4a20874ae9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } health_report { overall_health: UNKNOWN } } }
I20240501 07:59:09.256381 6172 catalog_manager.cc:5484] T 91a7e5059e2a497fa74b19733c8c6afc P 8b8b3ad3aa8f4053b2167dd550f2086f reported cstate change: term changed from 0 to 1, leader changed from <none> to 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129). New cstate: current_term: 1 leader_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } health_report { overall_health: UNKNOWN } } }
I20240501 07:59:09.260250 8794 raft_consensus.cc:1203] T 271e3c5cf9934eaead7fa1ba942084be P 74077b61d6f54342bf9736b4a9f2eade [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 07:59:09.262975 9676 tablet.cc:1722] T 9a17ba3dfb234636bcf63c3340aaab9d P 13d05ebdca8c4f489c597ce550dd773d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 07:59:09.263307 9676 tablet.cc:1620] T 9a17ba3dfb234636bcf63c3340aaab9d P 13d05ebdca8c4f489c597ce550dd773d: MemRowSet was empty: no flush needed.
I20240501 07:59:09.275075 6165 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55950:
table { table_name: "rebalance_test_table_6" } modify_external_catalogs: true
I20240501 07:59:09.275350 6165 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55950:
table { table_name: "rebalance_test_table_6" } modify_external_catalogs: true
I20240501 07:59:09.278280 9055 raft_consensus.cc:1203] T 782e37a4ebb34a68b461a74115ae8703 P a505e7cb133c4392bfda8a4a20874ae9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 07:59:09.281785 6165 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 0bc706e85ce74cef97a4827ea4d4aa65: Sending DeleteTablet for 4 replicas of tablet 26713549654249b09f820a31ee19e2b7
I20240501 07:59:09.282554 6294 tablet_service.cc:1514] Processing DeleteTablet for tablet 26713549654249b09f820a31ee19e2b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:09 UTC) from {username='slave'} at 127.0.0.1:45480
I20240501 07:59:09.282832 9388 tablet_replica.cc:330] T 26713549654249b09f820a31ee19e2b7 P 8b8b3ad3aa8f4053b2167dd550f2086f: stopping tablet replica
I20240501 07:59:09.282804 6686 tablet_service.cc:1514] Processing DeleteTablet for tablet 26713549654249b09f820a31ee19e2b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:09 UTC) from {username='slave'} at 127.0.0.1:38524
I20240501 07:59:09.282989 9388 raft_consensus.cc:2229] T 26713549654249b09f820a31ee19e2b7 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:09.283017 9387 tablet_replica.cc:330] T 26713549654249b09f820a31ee19e2b7 P 1f9cb72be98941b5a8e504cbe34132b9: stopping tablet replica
I20240501 07:59:09.283154 9387 raft_consensus.cc:2229] T 26713549654249b09f820a31ee19e2b7 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 LEADER]: Raft consensus shutting down.
I20240501 07:59:09.283155 9388 raft_consensus.cc:2258] T 26713549654249b09f820a31ee19e2b7 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:09.283370 9387 raft_consensus.cc:2258] T 26713549654249b09f820a31ee19e2b7 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:09.283736 9388 ts_tablet_manager.cc:1875] T 26713549654249b09f820a31ee19e2b7 P 8b8b3ad3aa8f4053b2167dd550f2086f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:59:09.283856 9387 ts_tablet_manager.cc:1875] T 26713549654249b09f820a31ee19e2b7 P 1f9cb72be98941b5a8e504cbe34132b9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:59:09.286027 9387 ts_tablet_manager.cc:1888] T 26713549654249b09f820a31ee19e2b7 P 1f9cb72be98941b5a8e504cbe34132b9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:59:09.286090 9387 log.cc:1196] T 26713549654249b09f820a31ee19e2b7 P 1f9cb72be98941b5a8e504cbe34132b9: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/26713549654249b09f820a31ee19e2b7
I20240501 07:59:09.286124 9388 ts_tablet_manager.cc:1888] T 26713549654249b09f820a31ee19e2b7 P 8b8b3ad3aa8f4053b2167dd550f2086f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:59:09.286197 9388 log.cc:1196] T 26713549654249b09f820a31ee19e2b7 P 8b8b3ad3aa8f4053b2167dd550f2086f: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/26713549654249b09f820a31ee19e2b7
I20240501 07:59:09.286338 9387 ts_tablet_manager.cc:1909] T 26713549654249b09f820a31ee19e2b7 P 1f9cb72be98941b5a8e504cbe34132b9: Deleting consensus metadata
I20240501 07:59:09.287443 6159 catalog_manager.cc:4830] TS 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132:36773): tablet 26713549654249b09f820a31ee19e2b7 (table rebalance_test_table_6 [id=09bfb3350ac042c892c1ef5bc6adf5e2]) successfully deleted
I20240501 07:59:09.288269 9388 ts_tablet_manager.cc:1909] T 26713549654249b09f820a31ee19e2b7 P 8b8b3ad3aa8f4053b2167dd550f2086f: Deleting consensus metadata
I20240501 07:59:09.289669 6424 tablet_service.cc:1514] Processing DeleteTablet for tablet 26713549654249b09f820a31ee19e2b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:09 UTC) from {username='slave'} at 127.0.0.1:45034
I20240501 07:59:09.289855 9036 tablet_service.cc:1514] Processing DeleteTablet for tablet 26713549654249b09f820a31ee19e2b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:09 UTC) from {username='slave'} at 127.0.0.1:44236
I20240501 07:59:09.290023 9841 tablet_replica.cc:330] T 26713549654249b09f820a31ee19e2b7 P a505e7cb133c4392bfda8a4a20874ae9: stopping tablet replica
I20240501 07:59:09.290165 9841 raft_consensus.cc:2229] T 26713549654249b09f820a31ee19e2b7 P a505e7cb133c4392bfda8a4a20874ae9 [term 1 LEARNER]: Raft consensus shutting down.
I20240501 07:59:09.290267 9841 raft_consensus.cc:2258] T 26713549654249b09f820a31ee19e2b7 P a505e7cb133c4392bfda8a4a20874ae9 [term 1 LEARNER]: Raft consensus is shut down!
I20240501 07:59:09.290575 6157 catalog_manager.cc:4830] TS 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129:46875): tablet 26713549654249b09f820a31ee19e2b7 (table rebalance_test_table_6 [id=09bfb3350ac042c892c1ef5bc6adf5e2]) successfully deleted
I20240501 07:59:09.290828 9841 ts_tablet_manager.cc:1875] T 26713549654249b09f820a31ee19e2b7 P a505e7cb133c4392bfda8a4a20874ae9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:59:09.290884 10075 tablet_replica.cc:330] T 26713549654249b09f820a31ee19e2b7 P 8315d22c56f7439a9e1fc82f153f018c: stopping tablet replica
I20240501 07:59:09.291016 10075 raft_consensus.cc:2229] T 26713549654249b09f820a31ee19e2b7 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:09.291155 10075 raft_consensus.cc:2258] T 26713549654249b09f820a31ee19e2b7 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:09.291611 10075 ts_tablet_manager.cc:1875] T 26713549654249b09f820a31ee19e2b7 P 8315d22c56f7439a9e1fc82f153f018c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:59:09.294071 10075 ts_tablet_manager.cc:1888] T 26713549654249b09f820a31ee19e2b7 P 8315d22c56f7439a9e1fc82f153f018c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:59:09.294200 10075 log.cc:1196] T 26713549654249b09f820a31ee19e2b7 P 8315d22c56f7439a9e1fc82f153f018c: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/26713549654249b09f820a31ee19e2b7
I20240501 07:59:09.294492 10075 ts_tablet_manager.cc:1909] T 26713549654249b09f820a31ee19e2b7 P 8315d22c56f7439a9e1fc82f153f018c: Deleting consensus metadata
I20240501 07:59:09.295480 6159 catalog_manager.cc:4830] TS 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130:32941): tablet 26713549654249b09f820a31ee19e2b7 (table rebalance_test_table_6 [id=09bfb3350ac042c892c1ef5bc6adf5e2]) successfully deleted
I20240501 07:59:09.295945 9841 ts_tablet_manager.cc:1888] T 26713549654249b09f820a31ee19e2b7 P a505e7cb133c4392bfda8a4a20874ae9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:59:09.296030 9841 log.cc:1196] T 26713549654249b09f820a31ee19e2b7 P a505e7cb133c4392bfda8a4a20874ae9: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/26713549654249b09f820a31ee19e2b7
I20240501 07:59:09.296329 9841 ts_tablet_manager.cc:1909] T 26713549654249b09f820a31ee19e2b7 P a505e7cb133c4392bfda8a4a20874ae9: Deleting consensus metadata
I20240501 07:59:09.297151 6158 catalog_manager.cc:4830] TS a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): tablet 26713549654249b09f820a31ee19e2b7 (table rebalance_test_table_6 [id=09bfb3350ac042c892c1ef5bc6adf5e2]) successfully deleted
I20240501 07:59:09.297200 6165 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 0bc706e85ce74cef97a4827ea4d4aa65: Sending DeleteTablet for 4 replicas of tablet ec2d3ccf0937430f947f3e2bde0506c5
I20240501 07:59:09.297837 6686 tablet_service.cc:1514] Processing DeleteTablet for tablet ec2d3ccf0937430f947f3e2bde0506c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:09 UTC) from {username='slave'} at 127.0.0.1:38524
I20240501 07:59:09.298028 9387 tablet_replica.cc:330] T ec2d3ccf0937430f947f3e2bde0506c5 P 1f9cb72be98941b5a8e504cbe34132b9: stopping tablet replica
I20240501 07:59:09.298190 9387 raft_consensus.cc:2229] T ec2d3ccf0937430f947f3e2bde0506c5 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:09.298241 6165 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 0bc706e85ce74cef97a4827ea4d4aa65: Sending DeleteTablet for 3 replicas of tablet e4c392558af14097b92f4cb35978eebe
I20240501 07:59:09.298383 9387 raft_consensus.cc:2258] T ec2d3ccf0937430f947f3e2bde0506c5 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:09.298908 9387 ts_tablet_manager.cc:1875] T ec2d3ccf0937430f947f3e2bde0506c5 P 1f9cb72be98941b5a8e504cbe34132b9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:59:09.299132 6424 tablet_service.cc:1514] Processing DeleteTablet for tablet e4c392558af14097b92f4cb35978eebe with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:09 UTC) from {username='slave'} at 127.0.0.1:45034
I20240501 07:59:09.299335 10075 tablet_replica.cc:330] T e4c392558af14097b92f4cb35978eebe P 8315d22c56f7439a9e1fc82f153f018c: stopping tablet replica
I20240501 07:59:09.299484 10075 raft_consensus.cc:2229] T e4c392558af14097b92f4cb35978eebe P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:09.299633 10075 raft_consensus.cc:2258] T e4c392558af14097b92f4cb35978eebe P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:09.299769 6294 tablet_service.cc:1514] Processing DeleteTablet for tablet ec2d3ccf0937430f947f3e2bde0506c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:09 UTC) from {username='slave'} at 127.0.0.1:45480
I20240501 07:59:09.299970 9388 tablet_replica.cc:330] T ec2d3ccf0937430f947f3e2bde0506c5 P 8b8b3ad3aa8f4053b2167dd550f2086f: stopping tablet replica
I20240501 07:59:09.300222 10075 ts_tablet_manager.cc:1875] T e4c392558af14097b92f4cb35978eebe P 8315d22c56f7439a9e1fc82f153f018c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:59:09.300297 9388 raft_consensus.cc:2229] T ec2d3ccf0937430f947f3e2bde0506c5 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 LEADER]: Raft consensus shutting down.
I20240501 07:59:09.300475 6556 tablet_service.cc:1514] Processing DeleteTablet for tablet ec2d3ccf0937430f947f3e2bde0506c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:09 UTC) from {username='slave'} at 127.0.0.1:53874
I20240501 07:59:09.300591 9388 raft_consensus.cc:2258] T ec2d3ccf0937430f947f3e2bde0506c5 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:09.301096 8905 tablet_service.cc:1514] Processing DeleteTablet for tablet ec2d3ccf0937430f947f3e2bde0506c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:09 UTC) from {username='slave'} at 127.0.0.1:47202
I20240501 07:59:09.301286 6295 tablet_service.cc:1514] Processing DeleteTablet for tablet e4c392558af14097b92f4cb35978eebe with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:09 UTC) from {username='slave'} at 127.0.0.1:45480
I20240501 07:59:09.301470 9840 tablet_replica.cc:330] T ec2d3ccf0937430f947f3e2bde0506c5 P 3b164cbbc00f4e8eb3254ddd349a3ff9: stopping tablet replica
I20240501 07:59:09.301607 9840 raft_consensus.cc:2229] T ec2d3ccf0937430f947f3e2bde0506c5 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:09.301751 9840 raft_consensus.cc:2258] T ec2d3ccf0937430f947f3e2bde0506c5 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:09.302291 9840 ts_tablet_manager.cc:1875] T ec2d3ccf0937430f947f3e2bde0506c5 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:59:09.302388 10075 ts_tablet_manager.cc:1888] T e4c392558af14097b92f4cb35978eebe P 8315d22c56f7439a9e1fc82f153f018c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 07:59:09.302453 10075 log.cc:1196] T e4c392558af14097b92f4cb35978eebe P 8315d22c56f7439a9e1fc82f153f018c: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/e4c392558af14097b92f4cb35978eebe
I20240501 07:59:09.302659 10075 ts_tablet_manager.cc:1909] T e4c392558af14097b92f4cb35978eebe P 8315d22c56f7439a9e1fc82f153f018c: Deleting consensus metadata
I20240501 07:59:09.302884 8794 raft_consensus.cc:1203] T 6dc929e2a6ae45bd8b1f297583636bb6 P 74077b61d6f54342bf9736b4a9f2eade [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 07:59:09.303570 6159 catalog_manager.cc:4830] TS 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130:32941): tablet e4c392558af14097b92f4cb35978eebe (table rebalance_test_table_6 [id=09bfb3350ac042c892c1ef5bc6adf5e2]) successfully deleted
I20240501 07:59:09.304463 9893 tablet_replica.cc:330] T ec2d3ccf0937430f947f3e2bde0506c5 P 13d05ebdca8c4f489c597ce550dd773d: stopping tablet replica
I20240501 07:59:09.304615 9893 raft_consensus.cc:2229] T ec2d3ccf0937430f947f3e2bde0506c5 P 13d05ebdca8c4f489c597ce550dd773d [term 1 LEARNER]: Raft consensus shutting down.
I20240501 07:59:09.304706 8924 raft_consensus.cc:1203] T 187940ba79f546b1ad79e7ec1de76442 P 13d05ebdca8c4f489c597ce550dd773d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 07:59:09.304718 9893 raft_consensus.cc:2258] T ec2d3ccf0937430f947f3e2bde0506c5 P 13d05ebdca8c4f489c597ce550dd773d [term 1 LEARNER]: Raft consensus is shut down!
I20240501 07:59:09.304901 6686 tablet_service.cc:1514] Processing DeleteTablet for tablet e4c392558af14097b92f4cb35978eebe with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:09 UTC) from {username='slave'} at 127.0.0.1:38524
I20240501 07:59:09.305923 9893 ts_tablet_manager.cc:1875] T ec2d3ccf0937430f947f3e2bde0506c5 P 13d05ebdca8c4f489c597ce550dd773d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:59:09.305359 9388 ts_tablet_manager.cc:1875] T ec2d3ccf0937430f947f3e2bde0506c5 P 8b8b3ad3aa8f4053b2167dd550f2086f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:59:09.308003 9893 ts_tablet_manager.cc:1888] T ec2d3ccf0937430f947f3e2bde0506c5 P 13d05ebdca8c4f489c597ce550dd773d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:59:09.308069 9893 log.cc:1196] T ec2d3ccf0937430f947f3e2bde0506c5 P 13d05ebdca8c4f489c597ce550dd773d: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/ec2d3ccf0937430f947f3e2bde0506c5
I20240501 07:59:09.308362 9893 ts_tablet_manager.cc:1909] T ec2d3ccf0937430f947f3e2bde0506c5 P 13d05ebdca8c4f489c597ce550dd773d: Deleting consensus metadata
I20240501 07:59:09.309057 6159 catalog_manager.cc:4830] TS 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134:40141): tablet ec2d3ccf0937430f947f3e2bde0506c5 (table rebalance_test_table_6 [id=09bfb3350ac042c892c1ef5bc6adf5e2]) successfully deleted
I20240501 07:59:09.309614 9388 ts_tablet_manager.cc:1888] T ec2d3ccf0937430f947f3e2bde0506c5 P 8b8b3ad3aa8f4053b2167dd550f2086f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:59:09.309706 9388 log.cc:1196] T ec2d3ccf0937430f947f3e2bde0506c5 P 8b8b3ad3aa8f4053b2167dd550f2086f: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/ec2d3ccf0937430f947f3e2bde0506c5
I20240501 07:59:09.309756 9840 ts_tablet_manager.cc:1888] T ec2d3ccf0937430f947f3e2bde0506c5 P 3b164cbbc00f4e8eb3254ddd349a3ff9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:59:09.309839 9840 log.cc:1196] T ec2d3ccf0937430f947f3e2bde0506c5 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/ec2d3ccf0937430f947f3e2bde0506c5
I20240501 07:59:09.310063 9388 ts_tablet_manager.cc:1909] T ec2d3ccf0937430f947f3e2bde0506c5 P 8b8b3ad3aa8f4053b2167dd550f2086f: Deleting consensus metadata
I20240501 07:59:09.310113 9840 ts_tablet_manager.cc:1909] T ec2d3ccf0937430f947f3e2bde0506c5 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Deleting consensus metadata
I20240501 07:59:09.310762 6157 catalog_manager.cc:4830] TS 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129:46875): tablet ec2d3ccf0937430f947f3e2bde0506c5 (table rebalance_test_table_6 [id=09bfb3350ac042c892c1ef5bc6adf5e2]) successfully deleted
I20240501 07:59:09.311110 6160 catalog_manager.cc:4830] TS 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131:34471): tablet ec2d3ccf0937430f947f3e2bde0506c5 (table rebalance_test_table_6 [id=09bfb3350ac042c892c1ef5bc6adf5e2]) successfully deleted
I20240501 07:59:09.311152 9388 tablet_replica.cc:330] T e4c392558af14097b92f4cb35978eebe P 8b8b3ad3aa8f4053b2167dd550f2086f: stopping tablet replica
I20240501 07:59:09.311331 9388 raft_consensus.cc:2229] T e4c392558af14097b92f4cb35978eebe P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:09.311470 9388 raft_consensus.cc:2258] T e4c392558af14097b92f4cb35978eebe P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:09.312055 9388 ts_tablet_manager.cc:1875] T e4c392558af14097b92f4cb35978eebe P 8b8b3ad3aa8f4053b2167dd550f2086f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:59:09.312215 9387 ts_tablet_manager.cc:1888] T ec2d3ccf0937430f947f3e2bde0506c5 P 1f9cb72be98941b5a8e504cbe34132b9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:59:09.312294 9387 log.cc:1196] T ec2d3ccf0937430f947f3e2bde0506c5 P 1f9cb72be98941b5a8e504cbe34132b9: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/ec2d3ccf0937430f947f3e2bde0506c5
I20240501 07:59:09.312565 9387 ts_tablet_manager.cc:1909] T ec2d3ccf0937430f947f3e2bde0506c5 P 1f9cb72be98941b5a8e504cbe34132b9: Deleting consensus metadata
I20240501 07:59:09.313241 9387 tablet_replica.cc:330] T e4c392558af14097b92f4cb35978eebe P 1f9cb72be98941b5a8e504cbe34132b9: stopping tablet replica
I20240501 07:59:09.313323 6159 catalog_manager.cc:4830] TS 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132:36773): tablet ec2d3ccf0937430f947f3e2bde0506c5 (table rebalance_test_table_6 [id=09bfb3350ac042c892c1ef5bc6adf5e2]) successfully deleted
I20240501 07:59:09.313452 9387 raft_consensus.cc:2229] T e4c392558af14097b92f4cb35978eebe P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 LEADER]: Raft consensus shutting down.
I20240501 07:59:09.313664 9387 raft_consensus.cc:2258] T e4c392558af14097b92f4cb35978eebe P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:09.315145 9388 ts_tablet_manager.cc:1888] T e4c392558af14097b92f4cb35978eebe P 8b8b3ad3aa8f4053b2167dd550f2086f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 07:59:09.315332 9388 log.cc:1196] T e4c392558af14097b92f4cb35978eebe P 8b8b3ad3aa8f4053b2167dd550f2086f: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/e4c392558af14097b92f4cb35978eebe
I20240501 07:59:09.315665 9388 ts_tablet_manager.cc:1909] T e4c392558af14097b92f4cb35978eebe P 8b8b3ad3aa8f4053b2167dd550f2086f: Deleting consensus metadata
I20240501 07:59:09.315745 9387 ts_tablet_manager.cc:1875] T e4c392558af14097b92f4cb35978eebe P 1f9cb72be98941b5a8e504cbe34132b9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:59:09.316388 6157 catalog_manager.cc:4830] TS 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129:46875): tablet e4c392558af14097b92f4cb35978eebe (table rebalance_test_table_6 [id=09bfb3350ac042c892c1ef5bc6adf5e2]) successfully deleted
I20240501 07:59:09.318764 9387 ts_tablet_manager.cc:1888] T e4c392558af14097b92f4cb35978eebe P 1f9cb72be98941b5a8e504cbe34132b9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 07:59:09.318851 9387 log.cc:1196] T e4c392558af14097b92f4cb35978eebe P 1f9cb72be98941b5a8e504cbe34132b9: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/e4c392558af14097b92f4cb35978eebe
I20240501 07:59:09.319146 9387 ts_tablet_manager.cc:1909] T e4c392558af14097b92f4cb35978eebe P 1f9cb72be98941b5a8e504cbe34132b9: Deleting consensus metadata
I20240501 07:59:09.319859 6159 catalog_manager.cc:4830] TS 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132:36773): tablet e4c392558af14097b92f4cb35978eebe (table rebalance_test_table_6 [id=09bfb3350ac042c892c1ef5bc6adf5e2]) successfully deleted
I20240501 07:59:09.332679 8757 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:59:09.339138 6556 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 07:59:09.339982 6686 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 07:59:09.340792 9167 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:59:09.341399 6424 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 07:59:09.345796 6164 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55950:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 07:59:09.349884 6295 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 07:59:09.351567 9036 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:59:09.358937 9675 tablet.cc:1722] T 9a17ba3dfb234636bcf63c3340aaab9d P 74077b61d6f54342bf9736b4a9f2eade: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 07:59:09.359316 9675 tablet.cc:1620] T 9a17ba3dfb234636bcf63c3340aaab9d P 74077b61d6f54342bf9736b4a9f2eade: MemRowSet was empty: no flush needed.
I20240501 07:59:09.360177 9676 tablet.cc:1722] T 9a17ba3dfb234636bcf63c3340aaab9d P 13d05ebdca8c4f489c597ce550dd773d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 07:59:09.360512 9676 tablet.cc:1620] T 9a17ba3dfb234636bcf63c3340aaab9d P 13d05ebdca8c4f489c597ce550dd773d: MemRowSet was empty: no flush needed.
I20240501 07:59:09.367040 9672 tablet.cc:1722] T 9a17ba3dfb234636bcf63c3340aaab9d P 83f20e84af4c40d09a4804b6b26e697f: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 07:59:09.367391 9672 tablet.cc:1620] T 9a17ba3dfb234636bcf63c3340aaab9d P 83f20e84af4c40d09a4804b6b26e697f: MemRowSet was empty: no flush needed.
I20240501 07:59:09.368395 6172 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 0bc706e85ce74cef97a4827ea4d4aa65: rebalancer_dynamic_table [id=81aa8832269b4874bfec689c1e98d696] alter complete (version 2)
I20240501 07:59:09.372645 9055 raft_consensus.cc:1203] T 3b9b71d44f754353b2321e8c36dd4dfd P a505e7cb133c4392bfda8a4a20874ae9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 07:59:09.378414 9188 raft_consensus.cc:1203] T 404bb81b9c404dd288d2f8160a4273da P 83f20e84af4c40d09a4804b6b26e697f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 07:59:09.385629 8924 raft_consensus.cc:1203] T ff9dc384dbac47dc9795617c64688943 P 13d05ebdca8c4f489c597ce550dd773d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 07:59:09.390815 8924 raft_consensus.cc:1203] T bab34f083c8045e4a6df373db5ebd54b P 13d05ebdca8c4f489c597ce550dd773d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 07:59:09.401269 8905 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:59:09.412994 9188 raft_consensus.cc:1203] T 6e2ac8c8857d45b88d3b51147d2609aa P 83f20e84af4c40d09a4804b6b26e697f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 07:59:09.420039 8794 raft_consensus.cc:1203] T 5ee738571af740c1b494d4f17413d1d8 P 74077b61d6f54342bf9736b4a9f2eade [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 07:59:09.424019 9188 raft_consensus.cc:1203] T 7d51b1940b594c79a7d13d7026b76da8 P 83f20e84af4c40d09a4804b6b26e697f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 07:59:09.466918 6575 consensus_queue.cc:237] T 3745acee2b5543b697a3c8f45d2618bb P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: true } }
I20240501 07:59:09.468706 6444 raft_consensus.cc:1261] T 3745acee2b5543b697a3c8f45d2618bb P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Refusing update from remote peer 3b164cbbc00f4e8eb3254ddd349a3ff9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:59:09.468701 6314 raft_consensus.cc:1261] T 3745acee2b5543b697a3c8f45d2618bb P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Refusing update from remote peer 3b164cbbc00f4e8eb3254ddd349a3ff9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:59:09.469198 8680 consensus_queue.cc:1035] T 3745acee2b5543b697a3c8f45d2618bb P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:09.469864 8680 consensus_queue.cc:1035] T 3745acee2b5543b697a3c8f45d2618bb P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:09.471760 10046 raft_consensus.cc:2932] T 3745acee2b5543b697a3c8f45d2618bb P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: true } } }
I20240501 07:59:09.472151 6314 raft_consensus.cc:2932] T 3745acee2b5543b697a3c8f45d2618bb P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: true } } }
I20240501 07:59:09.473172 6444 raft_consensus.cc:2932] T 3745acee2b5543b697a3c8f45d2618bb P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: true } } }
I20240501 07:59:09.474521 6164 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55950:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 07:59:09.475119 6172 catalog_manager.cc:5484] T 3745acee2b5543b697a3c8f45d2618bb P 3b164cbbc00f4e8eb3254ddd349a3ff9 reported cstate change: config changed from index -1 to 4, NON_VOTER a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240501 07:59:09.477779 6511 consensus_peers.cc:485] T 3745acee2b5543b697a3c8f45d2618bb P 3b164cbbc00f4e8eb3254ddd349a3ff9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3745acee2b5543b697a3c8f45d2618bb. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:09.481325 9675 tablet.cc:1722] T 9a17ba3dfb234636bcf63c3340aaab9d P 74077b61d6f54342bf9736b4a9f2eade: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 07:59:09.482479 9672 tablet.cc:1722] T 9a17ba3dfb234636bcf63c3340aaab9d P 83f20e84af4c40d09a4804b6b26e697f: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 07:59:09.484601 9676 tablet.cc:1722] T 9a17ba3dfb234636bcf63c3340aaab9d P 13d05ebdca8c4f489c597ce550dd773d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 07:59:09.485569 8794 raft_consensus.cc:1203] T c3a9036ccfee45498bca9681518d3699 P 74077b61d6f54342bf9736b4a9f2eade [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 07:59:09.489169 6172 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 0bc706e85ce74cef97a4827ea4d4aa65: rebalancer_dynamic_table_new_name [id=81aa8832269b4874bfec689c1e98d696] alter complete (version 3)
I20240501 07:59:09.495731 6708 consensus_queue.cc:237] T 72feac2a993141168ac9399d9361ef17 P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: true } }
I20240501 07:59:09.497116 6575 raft_consensus.cc:1261] T 72feac2a993141168ac9399d9361ef17 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Refusing update from remote peer 1f9cb72be98941b5a8e504cbe34132b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 07:59:09.497364 6643 consensus_peers.cc:485] T 72feac2a993141168ac9399d9361ef17 P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134:40141): Couldn't send request to peer 13d05ebdca8c4f489c597ce550dd773d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 72feac2a993141168ac9399d9361ef17. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:09.497577 6444 raft_consensus.cc:1261] T 72feac2a993141168ac9399d9361ef17 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Refusing update from remote peer 1f9cb72be98941b5a8e504cbe34132b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:59:09.497545 9398 consensus_queue.cc:1035] T 72feac2a993141168ac9399d9361ef17 P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:09.497960 9398 consensus_queue.cc:1035] T 72feac2a993141168ac9399d9361ef17 P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:09.499581 9990 raft_consensus.cc:2932] T 72feac2a993141168ac9399d9361ef17 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: true } } }
I20240501 07:59:09.500012 6575 raft_consensus.cc:2932] T 72feac2a993141168ac9399d9361ef17 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: true } } }
I20240501 07:59:09.500038 6444 raft_consensus.cc:2932] T 72feac2a993141168ac9399d9361ef17 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: true } } }
I20240501 07:59:09.503182 6165 catalog_manager.cc:5484] T 72feac2a993141168ac9399d9361ef17 P 1f9cb72be98941b5a8e504cbe34132b9 reported cstate change: config changed from index -1 to 4, NON_VOTER 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "1f9cb72be98941b5a8e504cbe34132b9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 07:59:09.510923 6444 consensus_queue.cc:237] T f9df6f3df2b94e5eb97dbff04dbe2ae1 P 8315d22c56f7439a9e1fc82f153f018c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { replace: true } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: true } }
I20240501 07:59:09.512271 6708 raft_consensus.cc:1261] T f9df6f3df2b94e5eb97dbff04dbe2ae1 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Refusing update from remote peer 8315d22c56f7439a9e1fc82f153f018c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:59:09.512281 6314 raft_consensus.cc:1261] T f9df6f3df2b94e5eb97dbff04dbe2ae1 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Refusing update from remote peer 8315d22c56f7439a9e1fc82f153f018c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 07:59:09.512583 6380 consensus_peers.cc:485] T f9df6f3df2b94e5eb97dbff04dbe2ae1 P 8315d22c56f7439a9e1fc82f153f018c -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f9df6f3df2b94e5eb97dbff04dbe2ae1. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:09.512735 9489 consensus_queue.cc:1035] T f9df6f3df2b94e5eb97dbff04dbe2ae1 P 8315d22c56f7439a9e1fc82f153f018c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:09.512920 8512 consensus_queue.cc:1035] T f9df6f3df2b94e5eb97dbff04dbe2ae1 P 8315d22c56f7439a9e1fc82f153f018c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:09.515565 9489 raft_consensus.cc:2932] T f9df6f3df2b94e5eb97dbff04dbe2ae1 P 8315d22c56f7439a9e1fc82f153f018c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { replace: true } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: true } } }
I20240501 07:59:09.516206 6708 raft_consensus.cc:2932] T f9df6f3df2b94e5eb97dbff04dbe2ae1 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { replace: true } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: true } } }
I20240501 07:59:09.517371 6313 raft_consensus.cc:2932] T f9df6f3df2b94e5eb97dbff04dbe2ae1 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { replace: true } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: true } } }
I20240501 07:59:09.518664 6164 catalog_manager.cc:5484] T f9df6f3df2b94e5eb97dbff04dbe2ae1 P 8315d22c56f7439a9e1fc82f153f018c reported cstate change: config changed from index -1 to 4, NON_VOTER 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "8315d22c56f7439a9e1fc82f153f018c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 07:59:09.563530 8794 raft_consensus.cc:1203] T b7d57b7fd5234f50afbc859bc997f3e1 P 74077b61d6f54342bf9736b4a9f2eade [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 07:59:09.566490 9055 raft_consensus.cc:1203] T 8454f24b0c28451b8ee51be39ca3e07f P a505e7cb133c4392bfda8a4a20874ae9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 07:59:09.591831 6167 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55950:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 07:59:09.591969 6167 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55950:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 07:59:09.594668 6167 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 0bc706e85ce74cef97a4827ea4d4aa65: Sending DeleteTablet for 3 replicas of tablet 9a17ba3dfb234636bcf63c3340aaab9d
I20240501 07:59:09.595196 8905 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a17ba3dfb234636bcf63c3340aaab9d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:09 UTC) from {username='slave'} at 127.0.0.1:47202
I20240501 07:59:09.595309 8757 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a17ba3dfb234636bcf63c3340aaab9d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:09 UTC) from {username='slave'} at 127.0.0.1:43136
I20240501 07:59:09.595393 9893 tablet_replica.cc:330] T 9a17ba3dfb234636bcf63c3340aaab9d P 13d05ebdca8c4f489c597ce550dd773d: stopping tablet replica
I20240501 07:59:09.595402 9167 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a17ba3dfb234636bcf63c3340aaab9d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:09 UTC) from {username='slave'} at 127.0.0.1:33594
I20240501 07:59:09.595530 9893 raft_consensus.cc:2229] T 9a17ba3dfb234636bcf63c3340aaab9d P 13d05ebdca8c4f489c597ce550dd773d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:09.595644 9893 raft_consensus.cc:2258] T 9a17ba3dfb234636bcf63c3340aaab9d P 13d05ebdca8c4f489c597ce550dd773d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:09.595680 10174 tablet_replica.cc:330] T 9a17ba3dfb234636bcf63c3340aaab9d P 74077b61d6f54342bf9736b4a9f2eade: stopping tablet replica
I20240501 07:59:09.595825 10174 raft_consensus.cc:2229] T 9a17ba3dfb234636bcf63c3340aaab9d P 74077b61d6f54342bf9736b4a9f2eade [term 1 LEADER]: Raft consensus shutting down.
I20240501 07:59:09.596161 10174 raft_consensus.cc:2258] T 9a17ba3dfb234636bcf63c3340aaab9d P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:09.596304 9893 ts_tablet_manager.cc:1875] T 9a17ba3dfb234636bcf63c3340aaab9d P 13d05ebdca8c4f489c597ce550dd773d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:59:09.596486 10175 tablet_replica.cc:330] T 9a17ba3dfb234636bcf63c3340aaab9d P 83f20e84af4c40d09a4804b6b26e697f: stopping tablet replica
I20240501 07:59:09.596647 10175 raft_consensus.cc:2229] T 9a17ba3dfb234636bcf63c3340aaab9d P 83f20e84af4c40d09a4804b6b26e697f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:09.596779 10175 raft_consensus.cc:2258] T 9a17ba3dfb234636bcf63c3340aaab9d P 83f20e84af4c40d09a4804b6b26e697f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:09.597204 10174 ts_tablet_manager.cc:1875] T 9a17ba3dfb234636bcf63c3340aaab9d P 74077b61d6f54342bf9736b4a9f2eade: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:59:09.597352 10175 ts_tablet_manager.cc:1875] T 9a17ba3dfb234636bcf63c3340aaab9d P 83f20e84af4c40d09a4804b6b26e697f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:59:09.598416 9893 ts_tablet_manager.cc:1888] T 9a17ba3dfb234636bcf63c3340aaab9d P 13d05ebdca8c4f489c597ce550dd773d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:59:09.598472 9893 log.cc:1196] T 9a17ba3dfb234636bcf63c3340aaab9d P 13d05ebdca8c4f489c597ce550dd773d: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/9a17ba3dfb234636bcf63c3340aaab9d
I20240501 07:59:09.598714 9893 ts_tablet_manager.cc:1909] T 9a17ba3dfb234636bcf63c3340aaab9d P 13d05ebdca8c4f489c597ce550dd773d: Deleting consensus metadata
I20240501 07:59:09.599478 10174 ts_tablet_manager.cc:1888] T 9a17ba3dfb234636bcf63c3340aaab9d P 74077b61d6f54342bf9736b4a9f2eade: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:59:09.599478 10175 ts_tablet_manager.cc:1888] T 9a17ba3dfb234636bcf63c3340aaab9d P 83f20e84af4c40d09a4804b6b26e697f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:59:09.599560 10174 log.cc:1196] T 9a17ba3dfb234636bcf63c3340aaab9d P 74077b61d6f54342bf9736b4a9f2eade: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/9a17ba3dfb234636bcf63c3340aaab9d
I20240501 07:59:09.599509 6159 catalog_manager.cc:4830] TS 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134:40141): tablet 9a17ba3dfb234636bcf63c3340aaab9d (table rebalancer_dynamic_table_new_name [id=81aa8832269b4874bfec689c1e98d696]) successfully deleted
I20240501 07:59:09.599607 10175 log.cc:1196] T 9a17ba3dfb234636bcf63c3340aaab9d P 83f20e84af4c40d09a4804b6b26e697f: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/9a17ba3dfb234636bcf63c3340aaab9d
I20240501 07:59:09.599871 10174 ts_tablet_manager.cc:1909] T 9a17ba3dfb234636bcf63c3340aaab9d P 74077b61d6f54342bf9736b4a9f2eade: Deleting consensus metadata
I20240501 07:59:09.599923 10175 ts_tablet_manager.cc:1909] T 9a17ba3dfb234636bcf63c3340aaab9d P 83f20e84af4c40d09a4804b6b26e697f: Deleting consensus metadata
I20240501 07:59:09.600564 6157 catalog_manager.cc:4830] TS 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133:39859): tablet 9a17ba3dfb234636bcf63c3340aaab9d (table rebalancer_dynamic_table_new_name [id=81aa8832269b4874bfec689c1e98d696]) successfully deleted
I20240501 07:59:09.600564 6158 catalog_manager.cc:4830] TS 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): tablet 9a17ba3dfb234636bcf63c3340aaab9d (table rebalancer_dynamic_table_new_name [id=81aa8832269b4874bfec689c1e98d696]) successfully deleted
I20240501 07:59:09.680591 10111 raft_consensus.cc:1050] T 6dc929e2a6ae45bd8b1f297583636bb6 P 8315d22c56f7439a9e1fc82f153f018c: attempting to promote NON_VOTER 74077b61d6f54342bf9736b4a9f2eade to VOTER
I20240501 07:59:09.681159 10111 consensus_queue.cc:237] T 6dc929e2a6ae45bd8b1f297583636bb6 P 8315d22c56f7439a9e1fc82f153f018c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } }
I20240501 07:59:09.682806 6575 raft_consensus.cc:1261] T 6dc929e2a6ae45bd8b1f297583636bb6 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Refusing update from remote peer 8315d22c56f7439a9e1fc82f153f018c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:59:09.683149 8794 raft_consensus.cc:1261] T 6dc929e2a6ae45bd8b1f297583636bb6 P 74077b61d6f54342bf9736b4a9f2eade [term 1 LEARNER]: Refusing update from remote peer 8315d22c56f7439a9e1fc82f153f018c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:59:09.683327 10216 consensus_queue.cc:1035] T 6dc929e2a6ae45bd8b1f297583636bb6 P 8315d22c56f7439a9e1fc82f153f018c [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:59:09.683629 6313 raft_consensus.cc:1261] T 6dc929e2a6ae45bd8b1f297583636bb6 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Refusing update from remote peer 8315d22c56f7439a9e1fc82f153f018c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:59:09.683652 9116 consensus_queue.cc:1035] T 6dc929e2a6ae45bd8b1f297583636bb6 P 8315d22c56f7439a9e1fc82f153f018c [LEADER]: Connected to new peer: Peer: permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:59:09.684196 9116 consensus_queue.cc:1035] T 6dc929e2a6ae45bd8b1f297583636bb6 P 8315d22c56f7439a9e1fc82f153f018c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:59:09.686509 10216 raft_consensus.cc:2932] T 6dc929e2a6ae45bd8b1f297583636bb6 P 8315d22c56f7439a9e1fc82f153f018c [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } } }
I20240501 07:59:09.686993 6575 raft_consensus.cc:2932] T 6dc929e2a6ae45bd8b1f297583636bb6 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } } }
I20240501 07:59:09.688691 6313 raft_consensus.cc:2932] T 6dc929e2a6ae45bd8b1f297583636bb6 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } } }
I20240501 07:59:09.688686 8794 raft_consensus.cc:2932] T 6dc929e2a6ae45bd8b1f297583636bb6 P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } } }
I20240501 07:59:09.692354 6172 catalog_manager.cc:5484] T 6dc929e2a6ae45bd8b1f297583636bb6 P 74077b61d6f54342bf9736b4a9f2eade reported cstate change: config changed from index 4 to 5, 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8315d22c56f7439a9e1fc82f153f018c" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } } }
I20240501 07:59:09.693801 9964 consensus_queue.cc:1035] T 91a7e5059e2a497fa74b19733c8c6afc P 8b8b3ad3aa8f4053b2167dd550f2086f [LEADER]: Connected to new peer: Peer: permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:09.696708 6164 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55950:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 07:59:09.699429 6445 consensus_queue.cc:237] T 6dc929e2a6ae45bd8b1f297583636bb6 P 8315d22c56f7439a9e1fc82f153f018c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } }
I20240501 07:59:09.700753 10196 consensus_queue.cc:1035] T c86c420310ec42ffa5b24304fba74267 P 83f20e84af4c40d09a4804b6b26e697f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:09.700906 8794 raft_consensus.cc:1261] T 6dc929e2a6ae45bd8b1f297583636bb6 P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Refusing update from remote peer 8315d22c56f7439a9e1fc82f153f018c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 07:59:09.701486 6313 raft_consensus.cc:1261] T 6dc929e2a6ae45bd8b1f297583636bb6 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Refusing update from remote peer 8315d22c56f7439a9e1fc82f153f018c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 07:59:09.701948 8512 consensus_queue.cc:1035] T 6dc929e2a6ae45bd8b1f297583636bb6 P 8315d22c56f7439a9e1fc82f153f018c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 07:59:09.703815 8512 consensus_queue.cc:1035] T 6dc929e2a6ae45bd8b1f297583636bb6 P 8315d22c56f7439a9e1fc82f153f018c [LEADER]: Connected to new peer: Peer: permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 07:59:09.705125 8512 raft_consensus.cc:2932] T 6dc929e2a6ae45bd8b1f297583636bb6 P 8315d22c56f7439a9e1fc82f153f018c [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } } }
I20240501 07:59:09.705266 10040 consensus_queue.cc:1035] T 91a7e5059e2a497fa74b19733c8c6afc P 8b8b3ad3aa8f4053b2167dd550f2086f [LEADER]: Connected to new peer: Peer: permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:09.706846 9673 consensus_queue.cc:1035] T c86c420310ec42ffa5b24304fba74267 P 83f20e84af4c40d09a4804b6b26e697f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:09.706900 6159 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6dc929e2a6ae45bd8b1f297583636bb6 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 07:59:09.708022 10046 raft_consensus.cc:1050] T d3f2250a62c6429e886753aaffcbe883 P 3b164cbbc00f4e8eb3254ddd349a3ff9: attempting to promote NON_VOTER 83f20e84af4c40d09a4804b6b26e697f to VOTER
I20240501 07:59:09.708530 10046 consensus_queue.cc:237] T d3f2250a62c6429e886753aaffcbe883 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false } }
I20240501 07:59:09.710093 6444 raft_consensus.cc:1261] T d3f2250a62c6429e886753aaffcbe883 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Refusing update from remote peer 3b164cbbc00f4e8eb3254ddd349a3ff9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:59:09.710419 6313 raft_consensus.cc:2932] T 6dc929e2a6ae45bd8b1f297583636bb6 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } } }
I20240501 07:59:09.711131 6708 raft_consensus.cc:1261] T d3f2250a62c6429e886753aaffcbe883 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Refusing update from remote peer 3b164cbbc00f4e8eb3254ddd349a3ff9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:59:09.711563 9507 consensus_queue.cc:1035] T d3f2250a62c6429e886753aaffcbe883 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:59:09.712415 9507 consensus_queue.cc:1035] T d3f2250a62c6429e886753aaffcbe883 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:59:09.712393 6167 catalog_manager.cc:5484] T 6dc929e2a6ae45bd8b1f297583636bb6 P 8315d22c56f7439a9e1fc82f153f018c reported cstate change: config changed from index 5 to 6, VOTER 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "8315d22c56f7439a9e1fc82f153f018c" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 07:59:09.713464 9188 raft_consensus.cc:1261] T d3f2250a62c6429e886753aaffcbe883 P 83f20e84af4c40d09a4804b6b26e697f [term 1 LEARNER]: Refusing update from remote peer 3b164cbbc00f4e8eb3254ddd349a3ff9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:59:09.714640 9507 consensus_queue.cc:1035] T d3f2250a62c6429e886753aaffcbe883 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:59:09.715179 9167 tablet_service.cc:1467] Processing CreateTablet for tablet 51b9e79f78894a7fa240271f9a36f72b (DEFAULT_TABLE table=rebalancer_dynamic_table [id=4fa2f76148be476390669c4d79dbb6bd]), partition=
I20240501 07:59:09.715258 9036 tablet_service.cc:1467] Processing CreateTablet for tablet 51b9e79f78894a7fa240271f9a36f72b (DEFAULT_TABLE table=rebalancer_dynamic_table [id=4fa2f76148be476390669c4d79dbb6bd]), partition=
I20240501 07:59:09.715428 9167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51b9e79f78894a7fa240271f9a36f72b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:09.715507 9036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51b9e79f78894a7fa240271f9a36f72b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:09.716646 9507 raft_consensus.cc:2932] T d3f2250a62c6429e886753aaffcbe883 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false } } }
I20240501 07:59:09.717660 8905 tablet_service.cc:1467] Processing CreateTablet for tablet 51b9e79f78894a7fa240271f9a36f72b (DEFAULT_TABLE table=rebalancer_dynamic_table [id=4fa2f76148be476390669c4d79dbb6bd]), partition=
I20240501 07:59:09.717928 8905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51b9e79f78894a7fa240271f9a36f72b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:09.718994 6708 raft_consensus.cc:2932] T d3f2250a62c6429e886753aaffcbe883 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false } } }
I20240501 07:59:09.720073 6172 catalog_manager.cc:5484] T d3f2250a62c6429e886753aaffcbe883 P 3b164cbbc00f4e8eb3254ddd349a3ff9 reported cstate change: config changed from index 4 to 5, 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 07:59:09.722935 6444 raft_consensus.cc:2932] T d3f2250a62c6429e886753aaffcbe883 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false } } }
I20240501 07:59:09.724997 9940 consensus_queue.cc:1035] T 28eaf22342ff4b5394f958d0b659273c P a505e7cb133c4392bfda8a4a20874ae9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:09.725360 10260 tablet_bootstrap.cc:492] T 51b9e79f78894a7fa240271f9a36f72b P a505e7cb133c4392bfda8a4a20874ae9: Bootstrap starting.
I20240501 07:59:09.726760 10216 raft_consensus.cc:1050] T 271e3c5cf9934eaead7fa1ba942084be P 8315d22c56f7439a9e1fc82f153f018c: attempting to promote NON_VOTER 74077b61d6f54342bf9736b4a9f2eade to VOTER
I20240501 07:59:09.727149 10216 consensus_queue.cc:237] T 271e3c5cf9934eaead7fa1ba942084be P 8315d22c56f7439a9e1fc82f153f018c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } }
I20240501 07:59:09.727483 8793 raft_consensus.cc:2932] T 6dc929e2a6ae45bd8b1f297583636bb6 P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } } }
I20240501 07:59:09.727864 9188 raft_consensus.cc:2932] T d3f2250a62c6429e886753aaffcbe883 P 83f20e84af4c40d09a4804b6b26e697f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false } } }
I20240501 07:59:09.728543 6707 raft_consensus.cc:1261] T 271e3c5cf9934eaead7fa1ba942084be P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Refusing update from remote peer 8315d22c56f7439a9e1fc82f153f018c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:59:09.728770 6575 raft_consensus.cc:1261] T 271e3c5cf9934eaead7fa1ba942084be P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Refusing update from remote peer 8315d22c56f7439a9e1fc82f153f018c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:59:09.729038 10111 consensus_queue.cc:1035] T 271e3c5cf9934eaead7fa1ba942084be P 8315d22c56f7439a9e1fc82f153f018c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:59:09.729189 10216 consensus_queue.cc:1035] T 271e3c5cf9934eaead7fa1ba942084be P 8315d22c56f7439a9e1fc82f153f018c [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:59:09.731035 6556 tablet_service.cc:1514] Processing DeleteTablet for tablet 6dc929e2a6ae45bd8b1f297583636bb6 with delete_type TABLET_DATA_TOMBSTONED (TS 3b164cbbc00f4e8eb3254ddd349a3ff9 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:53874
I20240501 07:59:09.731241 9840 tablet_replica.cc:330] T 6dc929e2a6ae45bd8b1f297583636bb6 P 3b164cbbc00f4e8eb3254ddd349a3ff9: stopping tablet replica
I20240501 07:59:09.731402 9840 raft_consensus.cc:2229] T 6dc929e2a6ae45bd8b1f297583636bb6 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:09.731557 9840 raft_consensus.cc:2258] T 6dc929e2a6ae45bd8b1f297583636bb6 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:09.732435 9840 ts_tablet_manager.cc:1875] T 6dc929e2a6ae45bd8b1f297583636bb6 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:59:09.734395 8793 raft_consensus.cc:1261] T 271e3c5cf9934eaead7fa1ba942084be P 74077b61d6f54342bf9736b4a9f2eade [term 1 LEARNER]: Refusing update from remote peer 8315d22c56f7439a9e1fc82f153f018c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:59:09.734843 10111 consensus_queue.cc:1035] T 271e3c5cf9934eaead7fa1ba942084be P 8315d22c56f7439a9e1fc82f153f018c [LEADER]: Connected to new peer: Peer: permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:59:09.737391 10260 tablet_bootstrap.cc:654] T 51b9e79f78894a7fa240271f9a36f72b P a505e7cb133c4392bfda8a4a20874ae9: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:09.737391 6577 consensus_queue.cc:237] T d3f2250a62c6429e886753aaffcbe883 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false } }
I20240501 07:59:09.738905 6707 raft_consensus.cc:1261] T d3f2250a62c6429e886753aaffcbe883 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Refusing update from remote peer 3b164cbbc00f4e8eb3254ddd349a3ff9: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 07:59:09.739429 9840 ts_tablet_manager.cc:1888] T 6dc929e2a6ae45bd8b1f297583636bb6 P 3b164cbbc00f4e8eb3254ddd349a3ff9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 07:59:09.739962 9188 raft_consensus.cc:1261] T d3f2250a62c6429e886753aaffcbe883 P 83f20e84af4c40d09a4804b6b26e697f [term 1 FOLLOWER]: Refusing update from remote peer 3b164cbbc00f4e8eb3254ddd349a3ff9: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 07:59:09.739526 10046 consensus_queue.cc:1035] T d3f2250a62c6429e886753aaffcbe883 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 07:59:09.740433 10046 consensus_queue.cc:1035] T d3f2250a62c6429e886753aaffcbe883 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 07:59:09.741545 9255 consensus_queue.cc:1035] T 28eaf22342ff4b5394f958d0b659273c P a505e7cb133c4392bfda8a4a20874ae9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:09.742064 9840 log.cc:1196] T 6dc929e2a6ae45bd8b1f297583636bb6 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/6dc929e2a6ae45bd8b1f297583636bb6
I20240501 07:59:09.742602 6160 catalog_manager.cc:4830] TS 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131:34471): tablet 6dc929e2a6ae45bd8b1f297583636bb6 (table rebalance_test_table_9 [id=c4f46595186f4427aa00d6752aa5dc9a]) successfully deleted
I20240501 07:59:09.743803 10046 raft_consensus.cc:2932] T d3f2250a62c6429e886753aaffcbe883 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false } } }
I20240501 07:59:09.744544 10260 tablet_bootstrap.cc:492] T 51b9e79f78894a7fa240271f9a36f72b P a505e7cb133c4392bfda8a4a20874ae9: No bootstrap required, opened a new log
I20240501 07:59:09.744633 10260 ts_tablet_manager.cc:1367] T 51b9e79f78894a7fa240271f9a36f72b P a505e7cb133c4392bfda8a4a20874ae9: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.000s
I20240501 07:59:09.744536 6707 raft_consensus.cc:2932] T d3f2250a62c6429e886753aaffcbe883 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false } } }
I20240501 07:59:09.745122 10260 raft_consensus.cc:348] T 51b9e79f78894a7fa240271f9a36f72b P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } }
I20240501 07:59:09.745220 10260 raft_consensus.cc:374] T 51b9e79f78894a7fa240271f9a36f72b P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:09.745249 10260 raft_consensus.cc:729] T 51b9e79f78894a7fa240271f9a36f72b P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a505e7cb133c4392bfda8a4a20874ae9, State: Initialized, Role: FOLLOWER
I20240501 07:59:09.745399 10260 consensus_queue.cc:260] T 51b9e79f78894a7fa240271f9a36f72b P a505e7cb133c4392bfda8a4a20874ae9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } }
I20240501 07:59:09.745817 6160 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d3f2250a62c6429e886753aaffcbe883 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 07:59:09.746604 10111 raft_consensus.cc:2932] T 271e3c5cf9934eaead7fa1ba942084be P 8315d22c56f7439a9e1fc82f153f018c [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } } }
I20240501 07:59:09.746991 9188 raft_consensus.cc:2932] T d3f2250a62c6429e886753aaffcbe883 P 83f20e84af4c40d09a4804b6b26e697f [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false } } }
I20240501 07:59:09.747674 6167 catalog_manager.cc:5484] T d3f2250a62c6429e886753aaffcbe883 P 3b164cbbc00f4e8eb3254ddd349a3ff9 reported cstate change: config changed from index 5 to 6, VOTER 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 07:59:09.748575 10260 ts_tablet_manager.cc:1398] T 51b9e79f78894a7fa240271f9a36f72b P a505e7cb133c4392bfda8a4a20874ae9: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240501 07:59:09.749979 6708 raft_consensus.cc:2932] T 271e3c5cf9934eaead7fa1ba942084be P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } } }
I20240501 07:59:09.750195 8793 raft_consensus.cc:2932] T 271e3c5cf9934eaead7fa1ba942084be P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } } }
I20240501 07:59:09.752382 6171 catalog_manager.cc:5484] T 271e3c5cf9934eaead7fa1ba942084be P 8315d22c56f7439a9e1fc82f153f018c reported cstate change: config changed from index 4 to 5, 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8315d22c56f7439a9e1fc82f153f018c" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 07:59:09.753445 10268 tablet_bootstrap.cc:492] T 51b9e79f78894a7fa240271f9a36f72b P 83f20e84af4c40d09a4804b6b26e697f: Bootstrap starting.
I20240501 07:59:09.753903 6575 raft_consensus.cc:2932] T 271e3c5cf9934eaead7fa1ba942084be P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } } }
I20240501 07:59:09.754467 10268 tablet_bootstrap.cc:654] T 51b9e79f78894a7fa240271f9a36f72b P 83f20e84af4c40d09a4804b6b26e697f: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:09.755124 10262 tablet_bootstrap.cc:492] T 51b9e79f78894a7fa240271f9a36f72b P 13d05ebdca8c4f489c597ce550dd773d: Bootstrap starting.
I20240501 07:59:09.755450 6424 tablet_service.cc:1514] Processing DeleteTablet for tablet d3f2250a62c6429e886753aaffcbe883 with delete_type TABLET_DATA_TOMBSTONED (TS 8315d22c56f7439a9e1fc82f153f018c not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:45034
I20240501 07:59:09.755681 10075 tablet_replica.cc:330] T d3f2250a62c6429e886753aaffcbe883 P 8315d22c56f7439a9e1fc82f153f018c: stopping tablet replica
I20240501 07:59:09.755815 10075 raft_consensus.cc:2229] T d3f2250a62c6429e886753aaffcbe883 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:09.755964 10075 raft_consensus.cc:2258] T d3f2250a62c6429e886753aaffcbe883 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:09.756116 10262 tablet_bootstrap.cc:654] T 51b9e79f78894a7fa240271f9a36f72b P 13d05ebdca8c4f489c597ce550dd773d: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:09.758244 10075 ts_tablet_manager.cc:1875] T d3f2250a62c6429e886753aaffcbe883 P 8315d22c56f7439a9e1fc82f153f018c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:59:09.758296 10268 tablet_bootstrap.cc:492] T 51b9e79f78894a7fa240271f9a36f72b P 83f20e84af4c40d09a4804b6b26e697f: No bootstrap required, opened a new log
I20240501 07:59:09.758381 10268 ts_tablet_manager.cc:1367] T 51b9e79f78894a7fa240271f9a36f72b P 83f20e84af4c40d09a4804b6b26e697f: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 07:59:09.758833 10268 raft_consensus.cc:348] T 51b9e79f78894a7fa240271f9a36f72b P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } }
I20240501 07:59:09.758936 10268 raft_consensus.cc:374] T 51b9e79f78894a7fa240271f9a36f72b P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:09.758965 10268 raft_consensus.cc:729] T 51b9e79f78894a7fa240271f9a36f72b P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83f20e84af4c40d09a4804b6b26e697f, State: Initialized, Role: FOLLOWER
I20240501 07:59:09.759680 10262 tablet_bootstrap.cc:492] T 51b9e79f78894a7fa240271f9a36f72b P 13d05ebdca8c4f489c597ce550dd773d: No bootstrap required, opened a new log
I20240501 07:59:09.759775 10262 ts_tablet_manager.cc:1367] T 51b9e79f78894a7fa240271f9a36f72b P 13d05ebdca8c4f489c597ce550dd773d: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 07:59:09.760561 10075 ts_tablet_manager.cc:1888] T d3f2250a62c6429e886753aaffcbe883 P 8315d22c56f7439a9e1fc82f153f018c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 07:59:09.760629 10075 log.cc:1196] T d3f2250a62c6429e886753aaffcbe883 P 8315d22c56f7439a9e1fc82f153f018c: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/d3f2250a62c6429e886753aaffcbe883
I20240501 07:59:09.762748 8680 raft_consensus.cc:1050] T ff9dc384dbac47dc9795617c64688943 P 3b164cbbc00f4e8eb3254ddd349a3ff9: attempting to promote NON_VOTER 13d05ebdca8c4f489c597ce550dd773d to VOTER
I20240501 07:59:09.763172 8680 consensus_queue.cc:237] T ff9dc384dbac47dc9795617c64688943 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false } }
I20240501 07:59:09.764379 10262 raft_consensus.cc:348] T 51b9e79f78894a7fa240271f9a36f72b P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } }
I20240501 07:59:09.764484 10262 raft_consensus.cc:374] T 51b9e79f78894a7fa240271f9a36f72b P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:09.764523 10262 raft_consensus.cc:729] T 51b9e79f78894a7fa240271f9a36f72b P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13d05ebdca8c4f489c597ce550dd773d, State: Initialized, Role: FOLLOWER
I20240501 07:59:09.764940 6313 raft_consensus.cc:1261] T ff9dc384dbac47dc9795617c64688943 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Refusing update from remote peer 3b164cbbc00f4e8eb3254ddd349a3ff9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:59:09.764822 10268 consensus_queue.cc:260] T 51b9e79f78894a7fa240271f9a36f72b P 83f20e84af4c40d09a4804b6b26e697f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } }
I20240501 07:59:09.765321 10268 ts_tablet_manager.cc:1398] T 51b9e79f78894a7fa240271f9a36f72b P 83f20e84af4c40d09a4804b6b26e697f: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20240501 07:59:09.764634 10262 consensus_queue.cc:260] T 51b9e79f78894a7fa240271f9a36f72b P 13d05ebdca8c4f489c597ce550dd773d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } }
I20240501 07:59:09.765895 9507 consensus_queue.cc:1035] T ff9dc384dbac47dc9795617c64688943 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:59:09.766072 8924 raft_consensus.cc:1261] T ff9dc384dbac47dc9795617c64688943 P 13d05ebdca8c4f489c597ce550dd773d [term 1 LEARNER]: Refusing update from remote peer 3b164cbbc00f4e8eb3254ddd349a3ff9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:59:09.767911 10262 ts_tablet_manager.cc:1398] T 51b9e79f78894a7fa240271f9a36f72b P 13d05ebdca8c4f489c597ce550dd773d: Time spent starting tablet: real 0.008s user 0.001s sys 0.000s
I20240501 07:59:09.768404 6708 raft_consensus.cc:1261] T ff9dc384dbac47dc9795617c64688943 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Refusing update from remote peer 3b164cbbc00f4e8eb3254ddd349a3ff9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:59:09.768596 6444 consensus_queue.cc:237] T 271e3c5cf9934eaead7fa1ba942084be P 8315d22c56f7439a9e1fc82f153f018c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } }
I20240501 07:59:09.769182 9507 consensus_queue.cc:1035] T ff9dc384dbac47dc9795617c64688943 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:59:09.769356 10046 consensus_queue.cc:1035] T ff9dc384dbac47dc9795617c64688943 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:59:09.769829 8793 raft_consensus.cc:1261] T 271e3c5cf9934eaead7fa1ba942084be P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Refusing update from remote peer 8315d22c56f7439a9e1fc82f153f018c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 07:59:09.769866 6708 raft_consensus.cc:1261] T 271e3c5cf9934eaead7fa1ba942084be P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Refusing update from remote peer 8315d22c56f7439a9e1fc82f153f018c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 07:59:09.770226 8512 consensus_queue.cc:1035] T 271e3c5cf9934eaead7fa1ba942084be P 8315d22c56f7439a9e1fc82f153f018c [LEADER]: Connected to new peer: Peer: permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 07:59:09.770290 6165 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55950:
name: "rebalancer_extra_table_6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 07:59:09.770567 8512 consensus_queue.cc:1035] T 271e3c5cf9934eaead7fa1ba942084be P 8315d22c56f7439a9e1fc82f153f018c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 07:59:09.772233 10216 raft_consensus.cc:2932] T 271e3c5cf9934eaead7fa1ba942084be P 8315d22c56f7439a9e1fc82f153f018c [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } } }
I20240501 07:59:09.774281 8793 raft_consensus.cc:2932] T 271e3c5cf9934eaead7fa1ba942084be P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } } }
I20240501 07:59:09.775071 6159 catalog_manager.cc:4830] TS 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130:32941): tablet d3f2250a62c6429e886753aaffcbe883 (table rebalance_test_table_13 [id=886e8147658c4cffb480ca3fb821c563]) successfully deleted
I20240501 07:59:09.775210 6159 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 271e3c5cf9934eaead7fa1ba942084be with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 07:59:09.776460 9398 raft_consensus.cc:1050] T 187940ba79f546b1ad79e7ec1de76442 P 1f9cb72be98941b5a8e504cbe34132b9: attempting to promote NON_VOTER 13d05ebdca8c4f489c597ce550dd773d to VOTER
I20240501 07:59:09.777037 9398 consensus_queue.cc:237] T 187940ba79f546b1ad79e7ec1de76442 P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { replace: true } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false } }
I20240501 07:59:09.777559 6172 catalog_manager.cc:5484] T 271e3c5cf9934eaead7fa1ba942084be P 74077b61d6f54342bf9736b4a9f2eade reported cstate change: config changed from index 5 to 6, VOTER 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "8315d22c56f7439a9e1fc82f153f018c" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } } }
I20240501 07:59:09.779057 6575 raft_consensus.cc:1261] T 187940ba79f546b1ad79e7ec1de76442 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Refusing update from remote peer 1f9cb72be98941b5a8e504cbe34132b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:59:09.779233 8925 raft_consensus.cc:1261] T 187940ba79f546b1ad79e7ec1de76442 P 13d05ebdca8c4f489c597ce550dd773d [term 1 LEARNER]: Refusing update from remote peer 1f9cb72be98941b5a8e504cbe34132b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:59:09.779448 9618 consensus_queue.cc:1035] T 187940ba79f546b1ad79e7ec1de76442 P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:59:09.781216 6314 raft_consensus.cc:1261] T 187940ba79f546b1ad79e7ec1de76442 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Refusing update from remote peer 1f9cb72be98941b5a8e504cbe34132b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:59:09.781713 9398 consensus_queue.cc:1035] T 187940ba79f546b1ad79e7ec1de76442 P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:59:09.781950 9990 consensus_queue.cc:1035] T 187940ba79f546b1ad79e7ec1de76442 P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:59:09.782541 6556 tablet_service.cc:1514] Processing DeleteTablet for tablet 271e3c5cf9934eaead7fa1ba942084be with delete_type TABLET_DATA_TOMBSTONED (TS 3b164cbbc00f4e8eb3254ddd349a3ff9 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:53874
I20240501 07:59:09.782745 9840 tablet_replica.cc:330] T 271e3c5cf9934eaead7fa1ba942084be P 3b164cbbc00f4e8eb3254ddd349a3ff9: stopping tablet replica
I20240501 07:59:09.782899 9840 raft_consensus.cc:2229] T 271e3c5cf9934eaead7fa1ba942084be P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:09.783061 9840 raft_consensus.cc:2258] T 271e3c5cf9934eaead7fa1ba942084be P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:09.783749 9036 tablet_service.cc:1467] Processing CreateTablet for tablet 239ee596e62d47169eeb6670dc62d59c (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=8b10e9a527644d40b1515561fc5df5c2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:09.784081 9036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 239ee596e62d47169eeb6670dc62d59c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:09.784535 9840 ts_tablet_manager.cc:1875] T 271e3c5cf9934eaead7fa1ba942084be P 3b164cbbc00f4e8eb3254ddd349a3ff9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:59:09.784581 8757 tablet_service.cc:1467] Processing CreateTablet for tablet 239ee596e62d47169eeb6670dc62d59c (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=8b10e9a527644d40b1515561fc5df5c2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:09.784852 8757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 239ee596e62d47169eeb6670dc62d59c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:09.786440 10260 tablet_bootstrap.cc:492] T 239ee596e62d47169eeb6670dc62d59c P a505e7cb133c4392bfda8a4a20874ae9: Bootstrap starting.
I20240501 07:59:09.786695 9840 ts_tablet_manager.cc:1888] T 271e3c5cf9934eaead7fa1ba942084be P 3b164cbbc00f4e8eb3254ddd349a3ff9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 07:59:09.786775 9840 log.cc:1196] T 271e3c5cf9934eaead7fa1ba942084be P 3b164cbbc00f4e8eb3254ddd349a3ff9: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/271e3c5cf9934eaead7fa1ba942084be
I20240501 07:59:09.787142 9167 tablet_service.cc:1467] Processing CreateTablet for tablet 2976d6353e8947b7b6c37f72efbeeac4 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=8b10e9a527644d40b1515561fc5df5c2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:09.787366 6160 catalog_manager.cc:4830] TS 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131:34471): tablet 271e3c5cf9934eaead7fa1ba942084be (table rebalance_test_table_15 [id=ecf5cb7235314f20bbb522737ba7fd3c]) successfully deleted
I20240501 07:59:09.787425 9167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2976d6353e8947b7b6c37f72efbeeac4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:09.788043 10260 tablet_bootstrap.cc:654] T 239ee596e62d47169eeb6670dc62d59c P a505e7cb133c4392bfda8a4a20874ae9: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:09.788048 10237 raft_consensus.cc:2932] T ff9dc384dbac47dc9795617c64688943 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false } } }
I20240501 07:59:09.788714 10293 tablet_bootstrap.cc:492] T 239ee596e62d47169eeb6670dc62d59c P 74077b61d6f54342bf9736b4a9f2eade: Bootstrap starting.
I20240501 07:59:09.789675 10293 tablet_bootstrap.cc:654] T 239ee596e62d47169eeb6670dc62d59c P 74077b61d6f54342bf9736b4a9f2eade: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:09.790475 6295 tablet_service.cc:1467] Processing CreateTablet for tablet 2f9e82c3f1df4d1292e37e9cea07186e (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=8b10e9a527644d40b1515561fc5df5c2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:09.790532 6294 tablet_service.cc:1467] Processing CreateTablet for tablet 2976d6353e8947b7b6c37f72efbeeac4 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=8b10e9a527644d40b1515561fc5df5c2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:09.790745 6295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f9e82c3f1df4d1292e37e9cea07186e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:09.790851 6294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2976d6353e8947b7b6c37f72efbeeac4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:09.790518 8924 raft_consensus.cc:2932] T ff9dc384dbac47dc9795617c64688943 P 13d05ebdca8c4f489c597ce550dd773d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false } } }
I20240501 07:59:09.793200 6708 raft_consensus.cc:2932] T ff9dc384dbac47dc9795617c64688943 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false } } }
I20240501 07:59:09.793630 10260 tablet_bootstrap.cc:492] T 239ee596e62d47169eeb6670dc62d59c P a505e7cb133c4392bfda8a4a20874ae9: No bootstrap required, opened a new log
I20240501 07:59:09.793716 10260 ts_tablet_manager.cc:1367] T 239ee596e62d47169eeb6670dc62d59c P a505e7cb133c4392bfda8a4a20874ae9: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240501 07:59:09.794157 10260 raft_consensus.cc:348] T 239ee596e62d47169eeb6670dc62d59c P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } }
I20240501 07:59:09.794263 10260 raft_consensus.cc:374] T 239ee596e62d47169eeb6670dc62d59c P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:09.794306 10260 raft_consensus.cc:729] T 239ee596e62d47169eeb6670dc62d59c P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a505e7cb133c4392bfda8a4a20874ae9, State: Initialized, Role: FOLLOWER
I20240501 07:59:09.794449 10260 consensus_queue.cc:260] T 239ee596e62d47169eeb6670dc62d59c P a505e7cb133c4392bfda8a4a20874ae9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } }
I20240501 07:59:09.794741 10260 ts_tablet_manager.cc:1398] T 239ee596e62d47169eeb6670dc62d59c P a505e7cb133c4392bfda8a4a20874ae9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:09.795286 9036 tablet_service.cc:1467] Processing CreateTablet for tablet 2976d6353e8947b7b6c37f72efbeeac4 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=8b10e9a527644d40b1515561fc5df5c2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:09.795538 9036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2976d6353e8947b7b6c37f72efbeeac4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:09.796687 6172 catalog_manager.cc:5484] T ff9dc384dbac47dc9795617c64688943 P 3b164cbbc00f4e8eb3254ddd349a3ff9 reported cstate change: config changed from index 4 to 5, 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 07:59:09.798925 10260 tablet_bootstrap.cc:492] T 2976d6353e8947b7b6c37f72efbeeac4 P a505e7cb133c4392bfda8a4a20874ae9: Bootstrap starting.
I20240501 07:59:09.799927 10260 tablet_bootstrap.cc:654] T 2976d6353e8947b7b6c37f72efbeeac4 P a505e7cb133c4392bfda8a4a20874ae9: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:09.800611 8757 tablet_service.cc:1467] Processing CreateTablet for tablet 2f9e82c3f1df4d1292e37e9cea07186e (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=8b10e9a527644d40b1515561fc5df5c2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:09.800899 8757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f9e82c3f1df4d1292e37e9cea07186e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:09.801259 10260 tablet_bootstrap.cc:492] T 2976d6353e8947b7b6c37f72efbeeac4 P a505e7cb133c4392bfda8a4a20874ae9: No bootstrap required, opened a new log
I20240501 07:59:09.801194 8905 tablet_service.cc:1467] Processing CreateTablet for tablet 239ee596e62d47169eeb6670dc62d59c (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=8b10e9a527644d40b1515561fc5df5c2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:09.801330 10260 ts_tablet_manager.cc:1367] T 2976d6353e8947b7b6c37f72efbeeac4 P a505e7cb133c4392bfda8a4a20874ae9: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 07:59:09.801461 8905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 239ee596e62d47169eeb6670dc62d59c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:09.801723 10260 raft_consensus.cc:348] T 2976d6353e8947b7b6c37f72efbeeac4 P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } }
I20240501 07:59:09.801823 10260 raft_consensus.cc:374] T 2976d6353e8947b7b6c37f72efbeeac4 P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:09.801856 10260 raft_consensus.cc:729] T 2976d6353e8947b7b6c37f72efbeeac4 P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a505e7cb133c4392bfda8a4a20874ae9, State: Initialized, Role: FOLLOWER
I20240501 07:59:09.801995 10260 consensus_queue.cc:260] T 2976d6353e8947b7b6c37f72efbeeac4 P a505e7cb133c4392bfda8a4a20874ae9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } }
I20240501 07:59:09.802263 10260 ts_tablet_manager.cc:1398] T 2976d6353e8947b7b6c37f72efbeeac4 P a505e7cb133c4392bfda8a4a20874ae9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:09.803573 10262 tablet_bootstrap.cc:492] T 239ee596e62d47169eeb6670dc62d59c P 13d05ebdca8c4f489c597ce550dd773d: Bootstrap starting.
I20240501 07:59:09.804419 10262 tablet_bootstrap.cc:654] T 239ee596e62d47169eeb6670dc62d59c P 13d05ebdca8c4f489c597ce550dd773d: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:09.804481 8512 raft_consensus.cc:1050] T bab34f083c8045e4a6df373db5ebd54b P 8315d22c56f7439a9e1fc82f153f018c: attempting to promote NON_VOTER 13d05ebdca8c4f489c597ce550dd773d to VOTER
I20240501 07:59:09.804893 8512 consensus_queue.cc:237] T bab34f083c8045e4a6df373db5ebd54b P 8315d22c56f7439a9e1fc82f153f018c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false } }
I20240501 07:59:09.806411 6707 raft_consensus.cc:2932] T 271e3c5cf9934eaead7fa1ba942084be P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } } }
I20240501 07:59:09.806597 8924 raft_consensus.cc:1261] T bab34f083c8045e4a6df373db5ebd54b P 13d05ebdca8c4f489c597ce550dd773d [term 1 LEARNER]: Refusing update from remote peer 8315d22c56f7439a9e1fc82f153f018c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:59:09.806635 6575 raft_consensus.cc:1261] T bab34f083c8045e4a6df373db5ebd54b P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Refusing update from remote peer 8315d22c56f7439a9e1fc82f153f018c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:59:09.807031 9489 consensus_queue.cc:1035] T bab34f083c8045e4a6df373db5ebd54b P 8315d22c56f7439a9e1fc82f153f018c [LEADER]: Connected to new peer: Peer: permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:59:09.807377 9116 consensus_queue.cc:1035] T bab34f083c8045e4a6df373db5ebd54b P 8315d22c56f7439a9e1fc82f153f018c [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:59:09.807606 6314 raft_consensus.cc:1261] T bab34f083c8045e4a6df373db5ebd54b P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Refusing update from remote peer 8315d22c56f7439a9e1fc82f153f018c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:59:09.808037 9489 consensus_queue.cc:1035] T bab34f083c8045e4a6df373db5ebd54b P 8315d22c56f7439a9e1fc82f153f018c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:59:09.810571 9116 raft_consensus.cc:2932] T bab34f083c8045e4a6df373db5ebd54b P 8315d22c56f7439a9e1fc82f153f018c [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false } } }
I20240501 07:59:09.811053 8905 tablet_service.cc:1467] Processing CreateTablet for tablet 2f9e82c3f1df4d1292e37e9cea07186e (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=8b10e9a527644d40b1515561fc5df5c2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:09.811332 8905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f9e82c3f1df4d1292e37e9cea07186e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:09.811502 9618 raft_consensus.cc:2932] T 187940ba79f546b1ad79e7ec1de76442 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { replace: true } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false } } }
I20240501 07:59:09.811751 6313 raft_consensus.cc:2932] T ff9dc384dbac47dc9795617c64688943 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false } } }
I20240501 07:59:09.812243 8924 raft_consensus.cc:2932] T 187940ba79f546b1ad79e7ec1de76442 P 13d05ebdca8c4f489c597ce550dd773d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { replace: true } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false } } }
I20240501 07:59:09.812350 8925 raft_consensus.cc:2932] T bab34f083c8045e4a6df373db5ebd54b P 13d05ebdca8c4f489c597ce550dd773d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false } } }
I20240501 07:59:09.813519 6575 raft_consensus.cc:2932] T 187940ba79f546b1ad79e7ec1de76442 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { replace: true } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false } } }
I20240501 07:59:09.814352 10262 tablet_bootstrap.cc:492] T 239ee596e62d47169eeb6670dc62d59c P 13d05ebdca8c4f489c597ce550dd773d: No bootstrap required, opened a new log
I20240501 07:59:09.814450 10262 ts_tablet_manager.cc:1367] T 239ee596e62d47169eeb6670dc62d59c P 13d05ebdca8c4f489c597ce550dd773d: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.000s
I20240501 07:59:09.814991 10262 raft_consensus.cc:348] T 239ee596e62d47169eeb6670dc62d59c P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } }
I20240501 07:59:09.815095 10262 raft_consensus.cc:374] T 239ee596e62d47169eeb6670dc62d59c P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:09.815131 10262 raft_consensus.cc:729] T 239ee596e62d47169eeb6670dc62d59c P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13d05ebdca8c4f489c597ce550dd773d, State: Initialized, Role: FOLLOWER
I20240501 07:59:09.815106 6577 raft_consensus.cc:2932] T bab34f083c8045e4a6df373db5ebd54b P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false } } }
I20240501 07:59:09.815244 10262 consensus_queue.cc:260] T 239ee596e62d47169eeb6670dc62d59c P 13d05ebdca8c4f489c597ce550dd773d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } }
I20240501 07:59:09.815531 10262 ts_tablet_manager.cc:1398] T 239ee596e62d47169eeb6670dc62d59c P 13d05ebdca8c4f489c597ce550dd773d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:09.815678 10262 tablet_bootstrap.cc:492] T 2f9e82c3f1df4d1292e37e9cea07186e P 13d05ebdca8c4f489c597ce550dd773d: Bootstrap starting.
I20240501 07:59:09.817965 10262 tablet_bootstrap.cc:654] T 2f9e82c3f1df4d1292e37e9cea07186e P 13d05ebdca8c4f489c597ce550dd773d: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:09.818406 6169 catalog_manager.cc:5484] T bab34f083c8045e4a6df373db5ebd54b P 8315d22c56f7439a9e1fc82f153f018c reported cstate change: config changed from index 4 to 5, 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8315d22c56f7439a9e1fc82f153f018c" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 07:59:09.818527 6313 raft_consensus.cc:2932] T 187940ba79f546b1ad79e7ec1de76442 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { replace: true } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false } } }
I20240501 07:59:09.819083 9489 raft_consensus.cc:1050] T 782e37a4ebb34a68b461a74115ae8703 P 8315d22c56f7439a9e1fc82f153f018c: attempting to promote NON_VOTER a505e7cb133c4392bfda8a4a20874ae9 to VOTER
I20240501 07:59:09.819497 9489 consensus_queue.cc:237] T 782e37a4ebb34a68b461a74115ae8703 P 8315d22c56f7439a9e1fc82f153f018c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false } }
I20240501 07:59:09.821660 6166 catalog_manager.cc:5484] T 187940ba79f546b1ad79e7ec1de76442 P 8b8b3ad3aa8f4053b2167dd550f2086f reported cstate change: config changed from index 4 to 5, 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1f9cb72be98941b5a8e504cbe34132b9" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { replace: true } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false } } }
I20240501 07:59:09.821907 6577 raft_consensus.cc:1261] T 782e37a4ebb34a68b461a74115ae8703 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Refusing update from remote peer 8315d22c56f7439a9e1fc82f153f018c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:59:09.822368 8512 consensus_queue.cc:1035] T 782e37a4ebb34a68b461a74115ae8703 P 8315d22c56f7439a9e1fc82f153f018c [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:59:09.823007 9055 raft_consensus.cc:1261] T 782e37a4ebb34a68b461a74115ae8703 P a505e7cb133c4392bfda8a4a20874ae9 [term 1 LEARNER]: Refusing update from remote peer 8315d22c56f7439a9e1fc82f153f018c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:59:09.823586 8512 consensus_queue.cc:1035] T 782e37a4ebb34a68b461a74115ae8703 P 8315d22c56f7439a9e1fc82f153f018c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:59:09.825018 10295 tablet_bootstrap.cc:492] T 2f9e82c3f1df4d1292e37e9cea07186e P 8b8b3ad3aa8f4053b2167dd550f2086f: Bootstrap starting.
I20240501 07:59:09.826400 6313 raft_consensus.cc:1261] T 782e37a4ebb34a68b461a74115ae8703 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Refusing update from remote peer 8315d22c56f7439a9e1fc82f153f018c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:59:09.826443 10295 tablet_bootstrap.cc:654] T 2f9e82c3f1df4d1292e37e9cea07186e P 8b8b3ad3aa8f4053b2167dd550f2086f: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:09.826767 9116 consensus_queue.cc:1035] T 782e37a4ebb34a68b461a74115ae8703 P 8315d22c56f7439a9e1fc82f153f018c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:59:09.827351 6170 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55950:
table { table_name: "rebalance_test_table_8" } modify_external_catalogs: true
I20240501 07:59:09.827463 6170 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55950:
table { table_name: "rebalance_test_table_8" } modify_external_catalogs: true
I20240501 07:59:09.829972 6170 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 0bc706e85ce74cef97a4827ea4d4aa65: Sending DeleteTablet for 3 replicas of tablet 12508ef4d9c942939ce08c14344754b4
I20240501 07:59:09.830333 6170 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 0bc706e85ce74cef97a4827ea4d4aa65: Sending DeleteTablet for 3 replicas of tablet 0cf117c67b524add96c9e2d887dc9e4d
I20240501 07:59:09.830435 6707 consensus_queue.cc:237] T 187940ba79f546b1ad79e7ec1de76442 P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false } }
I20240501 07:59:09.830648 6170 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 0bc706e85ce74cef97a4827ea4d4aa65: Sending DeleteTablet for 4 replicas of tablet 7d51b1940b594c79a7d13d7026b76da8
I20240501 07:59:09.830742 6314 raft_consensus.cc:2932] T bab34f083c8045e4a6df373db5ebd54b P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false } } }
I20240501 07:59:09.832032 9167 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d51b1940b594c79a7d13d7026b76da8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:09 UTC) from {username='slave'} at 127.0.0.1:33594
I20240501 07:59:09.832219 10175 tablet_replica.cc:330] T 7d51b1940b594c79a7d13d7026b76da8 P 83f20e84af4c40d09a4804b6b26e697f: stopping tablet replica
I20240501 07:59:09.832329 6556 tablet_service.cc:1514] Processing DeleteTablet for tablet 12508ef4d9c942939ce08c14344754b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:09 UTC) from {username='slave'} at 127.0.0.1:53874
I20240501 07:59:09.832376 10175 raft_consensus.cc:2229] T 7d51b1940b594c79a7d13d7026b76da8 P 83f20e84af4c40d09a4804b6b26e697f [term 1 LEARNER]: Raft consensus shutting down.
I20240501 07:59:09.832490 9840 tablet_replica.cc:330] T 12508ef4d9c942939ce08c14344754b4 P 3b164cbbc00f4e8eb3254ddd349a3ff9: stopping tablet replica
I20240501 07:59:09.832489 8924 raft_consensus.cc:1261] T 187940ba79f546b1ad79e7ec1de76442 P 13d05ebdca8c4f489c597ce550dd773d [term 1 FOLLOWER]: Refusing update from remote peer 1f9cb72be98941b5a8e504cbe34132b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 07:59:09.832490 10175 raft_consensus.cc:2258] T 7d51b1940b594c79a7d13d7026b76da8 P 83f20e84af4c40d09a4804b6b26e697f [term 1 LEARNER]: Raft consensus is shut down!
I20240501 07:59:09.832619 9840 raft_consensus.cc:2229] T 12508ef4d9c942939ce08c14344754b4 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:09.832787 9840 raft_consensus.cc:2258] T 12508ef4d9c942939ce08c14344754b4 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:09.832907 9990 consensus_queue.cc:1035] T 187940ba79f546b1ad79e7ec1de76442 P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 07:59:09.833089 6555 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d51b1940b594c79a7d13d7026b76da8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:09 UTC) from {username='slave'} at 127.0.0.1:53874
I20240501 07:59:09.833290 9840 ts_tablet_manager.cc:1875] T 12508ef4d9c942939ce08c14344754b4 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:59:09.833595 6577 raft_consensus.cc:1261] T 187940ba79f546b1ad79e7ec1de76442 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Refusing update from remote peer 1f9cb72be98941b5a8e504cbe34132b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 07:59:09.834136 6295 tablet_service.cc:1514] Processing DeleteTablet for tablet 12508ef4d9c942939ce08c14344754b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:09 UTC) from {username='slave'} at 127.0.0.1:45480
I20240501 07:59:09.834266 6293 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d51b1940b594c79a7d13d7026b76da8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:09 UTC) from {username='slave'} at 127.0.0.1:45480
I20240501 07:59:09.834648 9990 consensus_queue.cc:1035] T 187940ba79f546b1ad79e7ec1de76442 P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 07:59:09.834890 6424 tablet_service.cc:1514] Processing DeleteTablet for tablet 12508ef4d9c942939ce08c14344754b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:09 UTC) from {username='slave'} at 127.0.0.1:45034
I20240501 07:59:09.835052 10075 tablet_replica.cc:330] T 12508ef4d9c942939ce08c14344754b4 P 8315d22c56f7439a9e1fc82f153f018c: stopping tablet replica
I20240501 07:59:09.835122 6686 tablet_service.cc:1514] Processing DeleteTablet for tablet 0cf117c67b524add96c9e2d887dc9e4d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:09 UTC) from {username='slave'} at 127.0.0.1:38524
I20240501 07:59:09.835188 10075 raft_consensus.cc:2229] T 12508ef4d9c942939ce08c14344754b4 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:09.835307 6687 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d51b1940b594c79a7d13d7026b76da8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:09 UTC) from {username='slave'} at 127.0.0.1:38524
I20240501 07:59:09.835323 10075 raft_consensus.cc:2258] T 12508ef4d9c942939ce08c14344754b4 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:09.835765 6294 tablet_service.cc:1514] Processing DeleteTablet for tablet 0cf117c67b524add96c9e2d887dc9e4d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:09 UTC) from {username='slave'} at 127.0.0.1:45480
I20240501 07:59:09.835853 10075 ts_tablet_manager.cc:1875] T 12508ef4d9c942939ce08c14344754b4 P 8315d22c56f7439a9e1fc82f153f018c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:59:09.835901 6425 tablet_service.cc:1514] Processing DeleteTablet for tablet 0cf117c67b524add96c9e2d887dc9e4d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:09 UTC) from {username='slave'} at 127.0.0.1:45034
I20240501 07:59:09.836071 9840 ts_tablet_manager.cc:1888] T 12508ef4d9c942939ce08c14344754b4 P 3b164cbbc00f4e8eb3254ddd349a3ff9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 07:59:09.836128 9840 log.cc:1196] T 12508ef4d9c942939ce08c14344754b4 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/12508ef4d9c942939ce08c14344754b4
I20240501 07:59:09.836304 10175 ts_tablet_manager.cc:1875] T 7d51b1940b594c79a7d13d7026b76da8 P 83f20e84af4c40d09a4804b6b26e697f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:59:09.836408 9840 ts_tablet_manager.cc:1909] T 12508ef4d9c942939ce08c14344754b4 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Deleting consensus metadata
I20240501 07:59:09.836468 10268 tablet_bootstrap.cc:492] T 2976d6353e8947b7b6c37f72efbeeac4 P 83f20e84af4c40d09a4804b6b26e697f: Bootstrap starting.
I20240501 07:59:09.838006 10075 ts_tablet_manager.cc:1888] T 12508ef4d9c942939ce08c14344754b4 P 8315d22c56f7439a9e1fc82f153f018c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 07:59:09.838068 10075 log.cc:1196] T 12508ef4d9c942939ce08c14344754b4 P 8315d22c56f7439a9e1fc82f153f018c: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/12508ef4d9c942939ce08c14344754b4
I20240501 07:59:09.838330 10075 ts_tablet_manager.cc:1909] T 12508ef4d9c942939ce08c14344754b4 P 8315d22c56f7439a9e1fc82f153f018c: Deleting consensus metadata
I20240501 07:59:09.838547 10175 ts_tablet_manager.cc:1888] T 7d51b1940b594c79a7d13d7026b76da8 P 83f20e84af4c40d09a4804b6b26e697f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:59:09.838608 10175 log.cc:1196] T 7d51b1940b594c79a7d13d7026b76da8 P 83f20e84af4c40d09a4804b6b26e697f: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/7d51b1940b594c79a7d13d7026b76da8
I20240501 07:59:09.838864 10175 ts_tablet_manager.cc:1909] T 7d51b1940b594c79a7d13d7026b76da8 P 83f20e84af4c40d09a4804b6b26e697f: Deleting consensus metadata
I20240501 07:59:09.838886 10075 tablet_replica.cc:330] T 0cf117c67b524add96c9e2d887dc9e4d P 8315d22c56f7439a9e1fc82f153f018c: stopping tablet replica
I20240501 07:59:09.838990 10075 raft_consensus.cc:2229] T 0cf117c67b524add96c9e2d887dc9e4d P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:09.839110 10075 raft_consensus.cc:2258] T 0cf117c67b524add96c9e2d887dc9e4d P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:09.836948 9840 tablet_replica.cc:330] T 7d51b1940b594c79a7d13d7026b76da8 P 3b164cbbc00f4e8eb3254ddd349a3ff9: stopping tablet replica
I20240501 07:59:09.839396 6160 catalog_manager.cc:4830] TS 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131:34471): tablet 12508ef4d9c942939ce08c14344754b4 (table rebalance_test_table_8 [id=85a512f46553493eaf7ce4a1eecac55f]) successfully deleted
I20240501 07:59:09.839452 9840 raft_consensus.cc:2229] T 7d51b1940b594c79a7d13d7026b76da8 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:09.839571 10075 ts_tablet_manager.cc:1875] T 0cf117c67b524add96c9e2d887dc9e4d P 8315d22c56f7439a9e1fc82f153f018c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:59:09.839681 9840 raft_consensus.cc:2258] T 7d51b1940b594c79a7d13d7026b76da8 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:09.840139 9840 ts_tablet_manager.cc:1875] T 7d51b1940b594c79a7d13d7026b76da8 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:59:09.840524 10288 raft_consensus.cc:2932] T 187940ba79f546b1ad79e7ec1de76442 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false } } }
I20240501 07:59:09.840725 6158 catalog_manager.cc:4830] TS 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): tablet 7d51b1940b594c79a7d13d7026b76da8 (table rebalance_test_table_8 [id=85a512f46553493eaf7ce4a1eecac55f]) successfully deleted
I20240501 07:59:09.842188 9840 ts_tablet_manager.cc:1888] T 7d51b1940b594c79a7d13d7026b76da8 P 3b164cbbc00f4e8eb3254ddd349a3ff9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:59:09.842255 9840 log.cc:1196] T 7d51b1940b594c79a7d13d7026b76da8 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/7d51b1940b594c79a7d13d7026b76da8
I20240501 07:59:09.842528 9840 ts_tablet_manager.cc:1909] T 7d51b1940b594c79a7d13d7026b76da8 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Deleting consensus metadata
I20240501 07:59:09.843369 6160 catalog_manager.cc:4830] TS 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131:34471): tablet 7d51b1940b594c79a7d13d7026b76da8 (table rebalance_test_table_8 [id=85a512f46553493eaf7ce4a1eecac55f]) successfully deleted
I20240501 07:59:09.843379 6576 raft_consensus.cc:2932] T 187940ba79f546b1ad79e7ec1de76442 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false } } }
W20240501 07:59:09.845216 6643 consensus_peers.cc:485] T 0cf117c67b524add96c9e2d887dc9e4d P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130:32941): Couldn't send request to peer 8315d22c56f7439a9e1fc82f153f018c. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:09.845223 6172 catalog_manager.cc:5484] T 187940ba79f546b1ad79e7ec1de76442 P 1f9cb72be98941b5a8e504cbe34132b9 reported cstate change: config changed from index 5 to 6, VOTER 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "1f9cb72be98941b5a8e504cbe34132b9" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 07:59:09.846907 6159 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 187940ba79f546b1ad79e7ec1de76442 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 07:59:09.847366 10075 ts_tablet_manager.cc:1888] T 0cf117c67b524add96c9e2d887dc9e4d P 8315d22c56f7439a9e1fc82f153f018c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 07:59:09.847505 10075 log.cc:1196] T 0cf117c67b524add96c9e2d887dc9e4d P 8315d22c56f7439a9e1fc82f153f018c: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/0cf117c67b524add96c9e2d887dc9e4d
I20240501 07:59:09.847640 6294 tablet_service.cc:1514] Processing DeleteTablet for tablet 187940ba79f546b1ad79e7ec1de76442 with delete_type TABLET_DATA_TOMBSTONED (TS 8b8b3ad3aa8f4053b2167dd550f2086f not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:45480
I20240501 07:59:09.847038 6159 catalog_manager.cc:4830] TS 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130:32941): tablet 12508ef4d9c942939ce08c14344754b4 (table rebalance_test_table_8 [id=85a512f46553493eaf7ce4a1eecac55f]) successfully deleted
I20240501 07:59:09.848004 10075 ts_tablet_manager.cc:1909] T 0cf117c67b524add96c9e2d887dc9e4d P 8315d22c56f7439a9e1fc82f153f018c: Deleting consensus metadata
I20240501 07:59:09.848104 10310 tablet_replica.cc:330] T 0cf117c67b524add96c9e2d887dc9e4d P 1f9cb72be98941b5a8e504cbe34132b9: stopping tablet replica
I20240501 07:59:09.848214 10309 tablet_replica.cc:330] T 12508ef4d9c942939ce08c14344754b4 P 8b8b3ad3aa8f4053b2167dd550f2086f: stopping tablet replica
I20240501 07:59:09.848397 10310 raft_consensus.cc:2229] T 0cf117c67b524add96c9e2d887dc9e4d P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 LEADER]: Raft consensus shutting down.
I20240501 07:59:09.848570 10309 raft_consensus.cc:2229] T 12508ef4d9c942939ce08c14344754b4 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 LEADER]: Raft consensus shutting down.
I20240501 07:59:09.848802 10310 raft_consensus.cc:2258] T 0cf117c67b524add96c9e2d887dc9e4d P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:09.848882 8924 raft_consensus.cc:2932] T 187940ba79f546b1ad79e7ec1de76442 P 13d05ebdca8c4f489c597ce550dd773d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false } } }
I20240501 07:59:09.849409 10309 raft_consensus.cc:2258] T 12508ef4d9c942939ce08c14344754b4 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:09.849982 10309 ts_tablet_manager.cc:1875] T 12508ef4d9c942939ce08c14344754b4 P 8b8b3ad3aa8f4053b2167dd550f2086f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:59:09.850929 10268 tablet_bootstrap.cc:654] T 2976d6353e8947b7b6c37f72efbeeac4 P 83f20e84af4c40d09a4804b6b26e697f: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:09.851044 10310 ts_tablet_manager.cc:1875] T 0cf117c67b524add96c9e2d887dc9e4d P 1f9cb72be98941b5a8e504cbe34132b9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:59:09.851085 10293 tablet_bootstrap.cc:492] T 239ee596e62d47169eeb6670dc62d59c P 74077b61d6f54342bf9736b4a9f2eade: No bootstrap required, opened a new log
I20240501 07:59:09.851193 10293 ts_tablet_manager.cc:1367] T 239ee596e62d47169eeb6670dc62d59c P 74077b61d6f54342bf9736b4a9f2eade: Time spent bootstrapping tablet: real 0.063s user 0.002s sys 0.000s
I20240501 07:59:09.851661 10293 raft_consensus.cc:348] T 239ee596e62d47169eeb6670dc62d59c P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } }
I20240501 07:59:09.851769 10293 raft_consensus.cc:374] T 239ee596e62d47169eeb6670dc62d59c P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:09.851802 10293 raft_consensus.cc:729] T 239ee596e62d47169eeb6670dc62d59c P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74077b61d6f54342bf9736b4a9f2eade, State: Initialized, Role: FOLLOWER
I20240501 07:59:09.851922 10293 consensus_queue.cc:260] T 239ee596e62d47169eeb6670dc62d59c P 74077b61d6f54342bf9736b4a9f2eade [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } }
I20240501 07:59:09.852202 10293 ts_tablet_manager.cc:1398] T 239ee596e62d47169eeb6670dc62d59c P 74077b61d6f54342bf9736b4a9f2eade: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:09.852391 10293 tablet_bootstrap.cc:492] T 2f9e82c3f1df4d1292e37e9cea07186e P 74077b61d6f54342bf9736b4a9f2eade: Bootstrap starting.
I20240501 07:59:09.854081 10293 tablet_bootstrap.cc:654] T 2f9e82c3f1df4d1292e37e9cea07186e P 74077b61d6f54342bf9736b4a9f2eade: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:09.854908 6159 catalog_manager.cc:4830] TS 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130:32941): tablet 0cf117c67b524add96c9e2d887dc9e4d (table rebalance_test_table_8 [id=85a512f46553493eaf7ce4a1eecac55f]) successfully deleted
I20240501 07:59:09.855450 10216 raft_consensus.cc:2932] T 782e37a4ebb34a68b461a74115ae8703 P 8315d22c56f7439a9e1fc82f153f018c [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false } } }
I20240501 07:59:09.858268 10268 tablet_bootstrap.cc:492] T 2976d6353e8947b7b6c37f72efbeeac4 P 83f20e84af4c40d09a4804b6b26e697f: No bootstrap required, opened a new log
I20240501 07:59:09.858261 6575 raft_consensus.cc:2932] T 782e37a4ebb34a68b461a74115ae8703 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false } } }
I20240501 07:59:09.858408 10268 ts_tablet_manager.cc:1367] T 2976d6353e8947b7b6c37f72efbeeac4 P 83f20e84af4c40d09a4804b6b26e697f: Time spent bootstrapping tablet: real 0.022s user 0.002s sys 0.000s
I20240501 07:59:09.858634 6170 catalog_manager.cc:5484] T 782e37a4ebb34a68b461a74115ae8703 P 8315d22c56f7439a9e1fc82f153f018c reported cstate change: config changed from index 4 to 5, a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8315d22c56f7439a9e1fc82f153f018c" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240501 07:59:09.860075 6642 consensus_peers.cc:485] T 7d51b1940b594c79a7d13d7026b76da8 P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7d51b1940b594c79a7d13d7026b76da8. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:09.862357 10268 raft_consensus.cc:348] T 2976d6353e8947b7b6c37f72efbeeac4 P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } }
I20240501 07:59:09.862592 10268 raft_consensus.cc:374] T 2976d6353e8947b7b6c37f72efbeeac4 P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:09.862674 10268 raft_consensus.cc:729] T 2976d6353e8947b7b6c37f72efbeeac4 P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83f20e84af4c40d09a4804b6b26e697f, State: Initialized, Role: FOLLOWER
I20240501 07:59:09.862933 10268 consensus_queue.cc:260] T 2976d6353e8947b7b6c37f72efbeeac4 P 83f20e84af4c40d09a4804b6b26e697f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } }
I20240501 07:59:09.860647 9055 raft_consensus.cc:2932] T 782e37a4ebb34a68b461a74115ae8703 P a505e7cb133c4392bfda8a4a20874ae9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false } } }
I20240501 07:59:09.869192 10268 ts_tablet_manager.cc:1398] T 2976d6353e8947b7b6c37f72efbeeac4 P 83f20e84af4c40d09a4804b6b26e697f: Time spent starting tablet: real 0.011s user 0.001s sys 0.000s
I20240501 07:59:09.870236 6445 consensus_queue.cc:237] T 782e37a4ebb34a68b461a74115ae8703 P 8315d22c56f7439a9e1fc82f153f018c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false } }
I20240501 07:59:09.870515 10262 tablet_bootstrap.cc:492] T 2f9e82c3f1df4d1292e37e9cea07186e P 13d05ebdca8c4f489c597ce550dd773d: No bootstrap required, opened a new log
I20240501 07:59:09.870625 10262 ts_tablet_manager.cc:1367] T 2f9e82c3f1df4d1292e37e9cea07186e P 13d05ebdca8c4f489c597ce550dd773d: Time spent bootstrapping tablet: real 0.055s user 0.002s sys 0.000s
I20240501 07:59:09.871109 10262 raft_consensus.cc:348] T 2f9e82c3f1df4d1292e37e9cea07186e P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } }
I20240501 07:59:09.871229 10262 raft_consensus.cc:374] T 2f9e82c3f1df4d1292e37e9cea07186e P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:09.871281 10262 raft_consensus.cc:729] T 2f9e82c3f1df4d1292e37e9cea07186e P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13d05ebdca8c4f489c597ce550dd773d, State: Initialized, Role: FOLLOWER
I20240501 07:59:09.871439 9055 raft_consensus.cc:1261] T 782e37a4ebb34a68b461a74115ae8703 P a505e7cb133c4392bfda8a4a20874ae9 [term 1 FOLLOWER]: Refusing update from remote peer 8315d22c56f7439a9e1fc82f153f018c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 07:59:09.871423 10262 consensus_queue.cc:260] T 2f9e82c3f1df4d1292e37e9cea07186e P 13d05ebdca8c4f489c597ce550dd773d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } }
I20240501 07:59:09.871701 10262 ts_tablet_manager.cc:1398] T 2f9e82c3f1df4d1292e37e9cea07186e P 13d05ebdca8c4f489c597ce550dd773d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:09.871780 9489 consensus_queue.cc:1035] T 782e37a4ebb34a68b461a74115ae8703 P 8315d22c56f7439a9e1fc82f153f018c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 07:59:09.873585 10111 raft_consensus.cc:2932] T 782e37a4ebb34a68b461a74115ae8703 P 8315d22c56f7439a9e1fc82f153f018c [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false } } }
I20240501 07:59:09.875942 9055 raft_consensus.cc:2932] T 782e37a4ebb34a68b461a74115ae8703 P a505e7cb133c4392bfda8a4a20874ae9 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false } } }
I20240501 07:59:09.866035 6313 raft_consensus.cc:2932] T 782e37a4ebb34a68b461a74115ae8703 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false } } }
I20240501 07:59:09.866617 9373 raft_consensus.cc:1050] T 5ee738571af740c1b494d4f17413d1d8 P 8b8b3ad3aa8f4053b2167dd550f2086f: attempting to promote NON_VOTER 74077b61d6f54342bf9736b4a9f2eade to VOTER
I20240501 07:59:09.878531 6314 raft_consensus.cc:1261] T 782e37a4ebb34a68b461a74115ae8703 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Refusing update from remote peer 8315d22c56f7439a9e1fc82f153f018c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 07:59:09.868033 10295 tablet_bootstrap.cc:492] T 2f9e82c3f1df4d1292e37e9cea07186e P 8b8b3ad3aa8f4053b2167dd550f2086f: No bootstrap required, opened a new log
I20240501 07:59:09.878690 10295 ts_tablet_manager.cc:1367] T 2f9e82c3f1df4d1292e37e9cea07186e P 8b8b3ad3aa8f4053b2167dd550f2086f: Time spent bootstrapping tablet: real 0.054s user 0.002s sys 0.000s
I20240501 07:59:09.878922 9373 consensus_queue.cc:237] T 5ee738571af740c1b494d4f17413d1d8 P 8b8b3ad3aa8f4053b2167dd550f2086f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { replace: true } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } }
I20240501 07:59:09.879160 10295 raft_consensus.cc:348] T 2f9e82c3f1df4d1292e37e9cea07186e P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } }
I20240501 07:59:09.879279 10295 raft_consensus.cc:374] T 2f9e82c3f1df4d1292e37e9cea07186e P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:09.879312 10295 raft_consensus.cc:729] T 2f9e82c3f1df4d1292e37e9cea07186e P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b8b3ad3aa8f4053b2167dd550f2086f, State: Initialized, Role: FOLLOWER
I20240501 07:59:09.879415 10295 consensus_queue.cc:260] T 2f9e82c3f1df4d1292e37e9cea07186e P 8b8b3ad3aa8f4053b2167dd550f2086f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } }
I20240501 07:59:09.880678 6171 catalog_manager.cc:5484] T 782e37a4ebb34a68b461a74115ae8703 P 8315d22c56f7439a9e1fc82f153f018c reported cstate change: config changed from index 5 to 6, VOTER 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "8315d22c56f7439a9e1fc82f153f018c" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 07:59:09.881424 8793 raft_consensus.cc:1261] T 5ee738571af740c1b494d4f17413d1d8 P 74077b61d6f54342bf9736b4a9f2eade [term 1 LEARNER]: Refusing update from remote peer 8b8b3ad3aa8f4053b2167dd550f2086f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:59:09.881965 10295 ts_tablet_manager.cc:1398] T 2f9e82c3f1df4d1292e37e9cea07186e P 8b8b3ad3aa8f4053b2167dd550f2086f: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 07:59:09.882140 10295 tablet_bootstrap.cc:492] T 2976d6353e8947b7b6c37f72efbeeac4 P 8b8b3ad3aa8f4053b2167dd550f2086f: Bootstrap starting.
I20240501 07:59:09.882268 9489 consensus_queue.cc:1035] T 782e37a4ebb34a68b461a74115ae8703 P 8315d22c56f7439a9e1fc82f153f018c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 07:59:09.883159 10295 tablet_bootstrap.cc:654] T 2976d6353e8947b7b6c37f72efbeeac4 P 8b8b3ad3aa8f4053b2167dd550f2086f: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:09.883296 6575 raft_consensus.cc:1261] T 5ee738571af740c1b494d4f17413d1d8 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Refusing update from remote peer 8b8b3ad3aa8f4053b2167dd550f2086f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:59:09.883682 10040 consensus_queue.cc:1035] T 5ee738571af740c1b494d4f17413d1d8 P 8b8b3ad3aa8f4053b2167dd550f2086f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:59:09.883996 6707 raft_consensus.cc:1261] T 5ee738571af740c1b494d4f17413d1d8 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Refusing update from remote peer 8b8b3ad3aa8f4053b2167dd550f2086f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:59:09.884784 10040 consensus_queue.cc:1035] T 5ee738571af740c1b494d4f17413d1d8 P 8b8b3ad3aa8f4053b2167dd550f2086f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:59:09.885560 10040 consensus_queue.cc:1035] T 5ee738571af740c1b494d4f17413d1d8 P 8b8b3ad3aa8f4053b2167dd550f2086f [LEADER]: Connected to new peer: Peer: permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:59:09.886315 6555 tablet_service.cc:1514] Processing DeleteTablet for tablet 782e37a4ebb34a68b461a74115ae8703 with delete_type TABLET_DATA_TOMBSTONED (TS 3b164cbbc00f4e8eb3254ddd349a3ff9 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:53874
I20240501 07:59:09.886509 9840 tablet_replica.cc:330] T 782e37a4ebb34a68b461a74115ae8703 P 3b164cbbc00f4e8eb3254ddd349a3ff9: stopping tablet replica
I20240501 07:59:09.886639 9840 raft_consensus.cc:2229] T 782e37a4ebb34a68b461a74115ae8703 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:09.886778 9840 raft_consensus.cc:2258] T 782e37a4ebb34a68b461a74115ae8703 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:09.887383 9840 ts_tablet_manager.cc:1875] T 782e37a4ebb34a68b461a74115ae8703 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:59:09.887936 6314 raft_consensus.cc:2932] T 782e37a4ebb34a68b461a74115ae8703 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false } } }
I20240501 07:59:09.889779 9840 ts_tablet_manager.cc:1888] T 782e37a4ebb34a68b461a74115ae8703 P 3b164cbbc00f4e8eb3254ddd349a3ff9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 07:59:09.889855 9840 log.cc:1196] T 782e37a4ebb34a68b461a74115ae8703 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/782e37a4ebb34a68b461a74115ae8703
I20240501 07:59:09.889873 6159 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 782e37a4ebb34a68b461a74115ae8703 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 07:59:09.890376 6160 catalog_manager.cc:4830] TS 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131:34471): tablet 782e37a4ebb34a68b461a74115ae8703 (table rebalance_test_table_3 [id=eb6765f7643442ff82fde7f790b2aaed]) successfully deleted
I20240501 07:59:09.892657 10310 ts_tablet_manager.cc:1888] T 0cf117c67b524add96c9e2d887dc9e4d P 1f9cb72be98941b5a8e504cbe34132b9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 07:59:09.892745 10310 log.cc:1196] T 0cf117c67b524add96c9e2d887dc9e4d P 1f9cb72be98941b5a8e504cbe34132b9: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/0cf117c67b524add96c9e2d887dc9e4d
I20240501 07:59:09.894222 9964 raft_consensus.cc:2932] T 5ee738571af740c1b494d4f17413d1d8 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { replace: true } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } } }
I20240501 07:59:09.894614 6575 raft_consensus.cc:2932] T 5ee738571af740c1b494d4f17413d1d8 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { replace: true } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } } }
I20240501 07:59:09.896248 8793 raft_consensus.cc:2932] T 5ee738571af740c1b494d4f17413d1d8 P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { replace: true } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } } }
I20240501 07:59:09.897933 6164 catalog_manager.cc:5484] T 5ee738571af740c1b494d4f17413d1d8 P 3b164cbbc00f4e8eb3254ddd349a3ff9 reported cstate change: config changed from index 4 to 5, 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { replace: true } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } } }
I20240501 07:59:09.902401 6314 consensus_queue.cc:237] T 5ee738571af740c1b494d4f17413d1d8 P 8b8b3ad3aa8f4053b2167dd550f2086f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } }
I20240501 07:59:09.902891 8680 raft_consensus.cc:1050] T 6e2ac8c8857d45b88d3b51147d2609aa P 3b164cbbc00f4e8eb3254ddd349a3ff9: attempting to promote NON_VOTER 83f20e84af4c40d09a4804b6b26e697f to VOTER
I20240501 07:59:09.903280 8680 consensus_queue.cc:237] T 6e2ac8c8857d45b88d3b51147d2609aa P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false } }
I20240501 07:59:09.903717 8793 raft_consensus.cc:1261] T 5ee738571af740c1b494d4f17413d1d8 P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Refusing update from remote peer 8b8b3ad3aa8f4053b2167dd550f2086f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 07:59:09.904146 10040 consensus_queue.cc:1035] T 5ee738571af740c1b494d4f17413d1d8 P 8b8b3ad3aa8f4053b2167dd550f2086f [LEADER]: Connected to new peer: Peer: permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 07:59:09.904590 9188 raft_consensus.cc:1261] T 6e2ac8c8857d45b88d3b51147d2609aa P 83f20e84af4c40d09a4804b6b26e697f [term 1 LEARNER]: Refusing update from remote peer 3b164cbbc00f4e8eb3254ddd349a3ff9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:59:09.904932 6445 raft_consensus.cc:1261] T 6e2ac8c8857d45b88d3b51147d2609aa P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Refusing update from remote peer 3b164cbbc00f4e8eb3254ddd349a3ff9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:59:09.905400 10237 consensus_queue.cc:1035] T 6e2ac8c8857d45b88d3b51147d2609aa P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:59:09.905622 6575 raft_consensus.cc:1261] T 5ee738571af740c1b494d4f17413d1d8 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Refusing update from remote peer 8b8b3ad3aa8f4053b2167dd550f2086f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 07:59:09.906031 10040 consensus_queue.cc:1035] T 5ee738571af740c1b494d4f17413d1d8 P 8b8b3ad3aa8f4053b2167dd550f2086f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 07:59:09.906181 10237 consensus_queue.cc:1035] T 6e2ac8c8857d45b88d3b51147d2609aa P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:59:09.907903 10040 raft_consensus.cc:2932] T 5ee738571af740c1b494d4f17413d1d8 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } } }
I20240501 07:59:09.908063 6314 raft_consensus.cc:1261] T 6e2ac8c8857d45b88d3b51147d2609aa P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Refusing update from remote peer 3b164cbbc00f4e8eb3254ddd349a3ff9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:59:09.908386 10237 consensus_queue.cc:1035] T 6e2ac8c8857d45b88d3b51147d2609aa P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:59:09.894387 10309 ts_tablet_manager.cc:1888] T 12508ef4d9c942939ce08c14344754b4 P 8b8b3ad3aa8f4053b2167dd550f2086f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 07:59:09.910391 10309 log.cc:1196] T 12508ef4d9c942939ce08c14344754b4 P 8b8b3ad3aa8f4053b2167dd550f2086f: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/12508ef4d9c942939ce08c14344754b4
I20240501 07:59:09.910799 10309 ts_tablet_manager.cc:1909] T 12508ef4d9c942939ce08c14344754b4 P 8b8b3ad3aa8f4053b2167dd550f2086f: Deleting consensus metadata
I20240501 07:59:09.910720 6575 raft_consensus.cc:2932] T 5ee738571af740c1b494d4f17413d1d8 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } } }
I20240501 07:59:09.911412 10309 tablet_replica.cc:330] T 7d51b1940b594c79a7d13d7026b76da8 P 8b8b3ad3aa8f4053b2167dd550f2086f: stopping tablet replica
I20240501 07:59:09.911545 10309 raft_consensus.cc:2229] T 7d51b1940b594c79a7d13d7026b76da8 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:09.911690 10309 raft_consensus.cc:2258] T 7d51b1940b594c79a7d13d7026b76da8 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:09.912405 8680 raft_consensus.cc:2932] T 6e2ac8c8857d45b88d3b51147d2609aa P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false } } }
I20240501 07:59:09.913178 6157 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5ee738571af740c1b494d4f17413d1d8 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 07:59:09.913349 6157 catalog_manager.cc:4830] TS 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129:46875): tablet 12508ef4d9c942939ce08c14344754b4 (table rebalance_test_table_8 [id=85a512f46553493eaf7ce4a1eecac55f]) successfully deleted
I20240501 07:59:09.913561 8793 raft_consensus.cc:2932] T 5ee738571af740c1b494d4f17413d1d8 P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } } }
I20240501 07:59:09.913795 6171 catalog_manager.cc:5484] T 5ee738571af740c1b494d4f17413d1d8 P 3b164cbbc00f4e8eb3254ddd349a3ff9 reported cstate change: config changed from index 5 to 6, VOTER 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } } }
I20240501 07:59:09.913861 6445 raft_consensus.cc:2932] T 6e2ac8c8857d45b88d3b51147d2609aa P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false } } }
I20240501 07:59:09.917747 6170 catalog_manager.cc:5484] T 6e2ac8c8857d45b88d3b51147d2609aa P 8315d22c56f7439a9e1fc82f153f018c reported cstate change: config changed from index 4 to 5, 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false } } }
I20240501 07:59:09.918783 9188 raft_consensus.cc:2932] T 6e2ac8c8857d45b88d3b51147d2609aa P 83f20e84af4c40d09a4804b6b26e697f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false } } }
I20240501 07:59:09.921020 6575 consensus_queue.cc:237] T 6e2ac8c8857d45b88d3b51147d2609aa P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false } }
I20240501 07:59:09.921077 6314 raft_consensus.cc:2932] T 6e2ac8c8857d45b88d3b51147d2609aa P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false } } }
I20240501 07:59:09.922415 9188 raft_consensus.cc:1261] T 6e2ac8c8857d45b88d3b51147d2609aa P 83f20e84af4c40d09a4804b6b26e697f [term 1 FOLLOWER]: Refusing update from remote peer 3b164cbbc00f4e8eb3254ddd349a3ff9: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 07:59:09.922699 6313 raft_consensus.cc:1261] T 6e2ac8c8857d45b88d3b51147d2609aa P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Refusing update from remote peer 3b164cbbc00f4e8eb3254ddd349a3ff9: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 07:59:09.922868 10046 consensus_queue.cc:1035] T 6e2ac8c8857d45b88d3b51147d2609aa P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 07:59:09.923064 9507 consensus_queue.cc:1035] T 6e2ac8c8857d45b88d3b51147d2609aa P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 07:59:09.898198 6707 raft_consensus.cc:2932] T 5ee738571af740c1b494d4f17413d1d8 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { replace: true } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } } }
I20240501 07:59:09.923749 10310 ts_tablet_manager.cc:1909] T 0cf117c67b524add96c9e2d887dc9e4d P 1f9cb72be98941b5a8e504cbe34132b9: Deleting consensus metadata
I20240501 07:59:09.924436 10293 tablet_bootstrap.cc:492] T 2f9e82c3f1df4d1292e37e9cea07186e P 74077b61d6f54342bf9736b4a9f2eade: No bootstrap required, opened a new log
I20240501 07:59:09.924541 10293 ts_tablet_manager.cc:1367] T 2f9e82c3f1df4d1292e37e9cea07186e P 74077b61d6f54342bf9736b4a9f2eade: Time spent bootstrapping tablet: real 0.072s user 0.002s sys 0.000s
I20240501 07:59:09.924839 10046 raft_consensus.cc:2932] T 6e2ac8c8857d45b88d3b51147d2609aa P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false } } }
I20240501 07:59:09.925006 10293 raft_consensus.cc:348] T 2f9e82c3f1df4d1292e37e9cea07186e P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } }
I20240501 07:59:09.925110 10293 raft_consensus.cc:374] T 2f9e82c3f1df4d1292e37e9cea07186e P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:09.925141 10293 raft_consensus.cc:729] T 2f9e82c3f1df4d1292e37e9cea07186e P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74077b61d6f54342bf9736b4a9f2eade, State: Initialized, Role: FOLLOWER
I20240501 07:59:09.925189 6313 raft_consensus.cc:2932] T 6e2ac8c8857d45b88d3b51147d2609aa P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false } } }
I20240501 07:59:09.925277 10293 consensus_queue.cc:260] T 2f9e82c3f1df4d1292e37e9cea07186e P 74077b61d6f54342bf9736b4a9f2eade [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } }
I20240501 07:59:09.925593 10293 ts_tablet_manager.cc:1398] T 2f9e82c3f1df4d1292e37e9cea07186e P 74077b61d6f54342bf9736b4a9f2eade: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:09.926872 6160 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6e2ac8c8857d45b88d3b51147d2609aa with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 07:59:09.918810 6687 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ee738571af740c1b494d4f17413d1d8 with delete_type TABLET_DATA_TOMBSTONED (TS 1f9cb72be98941b5a8e504cbe34132b9 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:38524
I20240501 07:59:09.927261 9188 raft_consensus.cc:2932] T 6e2ac8c8857d45b88d3b51147d2609aa P 83f20e84af4c40d09a4804b6b26e697f [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false } } }
I20240501 07:59:09.927642 10310 tablet_replica.cc:330] T 7d51b1940b594c79a7d13d7026b76da8 P 1f9cb72be98941b5a8e504cbe34132b9: stopping tablet replica
I20240501 07:59:09.927745 6171 catalog_manager.cc:5484] T 6e2ac8c8857d45b88d3b51147d2609aa P 3b164cbbc00f4e8eb3254ddd349a3ff9 reported cstate change: config changed from index 5 to 6, VOTER 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 07:59:09.928398 10295 tablet_bootstrap.cc:492] T 2976d6353e8947b7b6c37f72efbeeac4 P 8b8b3ad3aa8f4053b2167dd550f2086f: No bootstrap required, opened a new log
I20240501 07:59:09.928534 10295 ts_tablet_manager.cc:1367] T 2976d6353e8947b7b6c37f72efbeeac4 P 8b8b3ad3aa8f4053b2167dd550f2086f: Time spent bootstrapping tablet: real 0.046s user 0.002s sys 0.000s
I20240501 07:59:09.928511 6159 catalog_manager.cc:4830] TS 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132:36773): tablet 0cf117c67b524add96c9e2d887dc9e4d (table rebalance_test_table_8 [id=85a512f46553493eaf7ce4a1eecac55f]) successfully deleted
I20240501 07:59:09.929080 10295 raft_consensus.cc:348] T 2976d6353e8947b7b6c37f72efbeeac4 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } }
I20240501 07:59:09.929250 10295 raft_consensus.cc:374] T 2976d6353e8947b7b6c37f72efbeeac4 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:09.929333 10295 raft_consensus.cc:729] T 2976d6353e8947b7b6c37f72efbeeac4 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b8b3ad3aa8f4053b2167dd550f2086f, State: Initialized, Role: FOLLOWER
I20240501 07:59:09.929510 10295 consensus_queue.cc:260] T 2976d6353e8947b7b6c37f72efbeeac4 P 8b8b3ad3aa8f4053b2167dd550f2086f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } }
I20240501 07:59:09.929891 10295 ts_tablet_manager.cc:1398] T 2976d6353e8947b7b6c37f72efbeeac4 P 8b8b3ad3aa8f4053b2167dd550f2086f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:09.930476 6425 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e2ac8c8857d45b88d3b51147d2609aa with delete_type TABLET_DATA_TOMBSTONED (TS 8315d22c56f7439a9e1fc82f153f018c not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:45034
I20240501 07:59:09.930734 10075 tablet_replica.cc:330] T 6e2ac8c8857d45b88d3b51147d2609aa P 8315d22c56f7439a9e1fc82f153f018c: stopping tablet replica
I20240501 07:59:09.930872 10075 raft_consensus.cc:2229] T 6e2ac8c8857d45b88d3b51147d2609aa P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:09.931021 10075 raft_consensus.cc:2258] T 6e2ac8c8857d45b88d3b51147d2609aa P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:09.931782 10075 ts_tablet_manager.cc:1875] T 6e2ac8c8857d45b88d3b51147d2609aa P 8315d22c56f7439a9e1fc82f153f018c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:59:09.934367 10075 ts_tablet_manager.cc:1888] T 6e2ac8c8857d45b88d3b51147d2609aa P 8315d22c56f7439a9e1fc82f153f018c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 07:59:09.934458 10075 log.cc:1196] T 6e2ac8c8857d45b88d3b51147d2609aa P 8315d22c56f7439a9e1fc82f153f018c: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/6e2ac8c8857d45b88d3b51147d2609aa
I20240501 07:59:09.935009 6159 catalog_manager.cc:4830] TS 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130:32941): tablet 6e2ac8c8857d45b88d3b51147d2609aa (table rebalance_test_table_17 [id=4de37f887eb14e2dbe5d1609df1e538f]) successfully deleted
I20240501 07:59:09.937974 10235 raft_consensus.cc:482] T 239ee596e62d47169eeb6670dc62d59c P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:09.938086 10235 raft_consensus.cc:504] T 239ee596e62d47169eeb6670dc62d59c P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } }
I20240501 07:59:09.938562 10235 leader_election.cc:288] T 239ee596e62d47169eeb6670dc62d59c P 13d05ebdca8c4f489c597ce550dd773d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133:39859), a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607)
I20240501 07:59:09.939077 8793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "239ee596e62d47169eeb6670dc62d59c" candidate_uuid: "13d05ebdca8c4f489c597ce550dd773d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74077b61d6f54342bf9736b4a9f2eade" is_pre_election: true
I20240501 07:59:09.939213 8793 raft_consensus.cc:2454] T 239ee596e62d47169eeb6670dc62d59c P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 13d05ebdca8c4f489c597ce550dd773d in term 0.
I20240501 07:59:09.939503 8858 leader_election.cc:302] T 239ee596e62d47169eeb6670dc62d59c P 13d05ebdca8c4f489c597ce550dd773d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13d05ebdca8c4f489c597ce550dd773d, 74077b61d6f54342bf9736b4a9f2eade; no voters:
I20240501 07:59:09.939709 10235 raft_consensus.cc:2781] T 239ee596e62d47169eeb6670dc62d59c P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:09.939767 10235 raft_consensus.cc:482] T 239ee596e62d47169eeb6670dc62d59c P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:09.939810 10235 raft_consensus.cc:3037] T 239ee596e62d47169eeb6670dc62d59c P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:09.940682 10235 raft_consensus.cc:504] T 239ee596e62d47169eeb6670dc62d59c P 13d05ebdca8c4f489c597ce550dd773d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } }
I20240501 07:59:09.941004 10235 leader_election.cc:288] T 239ee596e62d47169eeb6670dc62d59c P 13d05ebdca8c4f489c597ce550dd773d [CANDIDATE]: Term 1 election: Requested vote from peers 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133:39859), a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607)
I20240501 07:59:09.932440 9398 raft_consensus.cc:1050] T 404bb81b9c404dd288d2f8160a4273da P 1f9cb72be98941b5a8e504cbe34132b9: attempting to promote NON_VOTER 83f20e84af4c40d09a4804b6b26e697f to VOTER
I20240501 07:59:09.941107 10310 raft_consensus.cc:2229] T 7d51b1940b594c79a7d13d7026b76da8 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 LEADER]: Raft consensus shutting down.
I20240501 07:59:09.941474 9398 consensus_queue.cc:237] T 404bb81b9c404dd288d2f8160a4273da P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false } }
I20240501 07:59:09.941711 10309 ts_tablet_manager.cc:1875] T 7d51b1940b594c79a7d13d7026b76da8 P 8b8b3ad3aa8f4053b2167dd550f2086f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:59:09.932906 9990 raft_consensus.cc:1050] T 3b9b71d44f754353b2321e8c36dd4dfd P 1f9cb72be98941b5a8e504cbe34132b9: attempting to promote NON_VOTER a505e7cb133c4392bfda8a4a20874ae9 to VOTER
I20240501 07:59:09.943001 9990 consensus_queue.cc:237] T 3b9b71d44f754353b2321e8c36dd4dfd P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false } }
I20240501 07:59:09.943329 9056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "239ee596e62d47169eeb6670dc62d59c" candidate_uuid: "13d05ebdca8c4f489c597ce550dd773d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a505e7cb133c4392bfda8a4a20874ae9"
I20240501 07:59:09.943323 9055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "239ee596e62d47169eeb6670dc62d59c" candidate_uuid: "13d05ebdca8c4f489c597ce550dd773d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a505e7cb133c4392bfda8a4a20874ae9" is_pre_election: true
I20240501 07:59:09.943513 9056 raft_consensus.cc:3037] T 239ee596e62d47169eeb6670dc62d59c P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:09.944304 6445 raft_consensus.cc:1261] T 404bb81b9c404dd288d2f8160a4273da P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Refusing update from remote peer 1f9cb72be98941b5a8e504cbe34132b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:59:09.944412 9056 raft_consensus.cc:2454] T 239ee596e62d47169eeb6670dc62d59c P a505e7cb133c4392bfda8a4a20874ae9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13d05ebdca8c4f489c597ce550dd773d in term 1.
I20240501 07:59:09.944682 9188 raft_consensus.cc:1261] T 404bb81b9c404dd288d2f8160a4273da P 83f20e84af4c40d09a4804b6b26e697f [term 1 LEARNER]: Refusing update from remote peer 1f9cb72be98941b5a8e504cbe34132b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:59:09.944705 6313 raft_consensus.cc:1261] T 3b9b71d44f754353b2321e8c36dd4dfd P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Refusing update from remote peer 1f9cb72be98941b5a8e504cbe34132b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:59:09.944706 8859 leader_election.cc:302] T 239ee596e62d47169eeb6670dc62d59c P 13d05ebdca8c4f489c597ce550dd773d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13d05ebdca8c4f489c597ce550dd773d, a505e7cb133c4392bfda8a4a20874ae9; no voters:
I20240501 07:59:09.944862 10235 raft_consensus.cc:2781] T 239ee596e62d47169eeb6670dc62d59c P 13d05ebdca8c4f489c597ce550dd773d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:09.944933 10235 raft_consensus.cc:686] T 239ee596e62d47169eeb6670dc62d59c P 13d05ebdca8c4f489c597ce550dd773d [term 1 LEADER]: Becoming Leader. State: Replica: 13d05ebdca8c4f489c597ce550dd773d, State: Running, Role: LEADER
I20240501 07:59:09.945034 9618 consensus_queue.cc:1035] T 404bb81b9c404dd288d2f8160a4273da P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:59:09.945098 10235 consensus_queue.cc:237] T 239ee596e62d47169eeb6670dc62d59c P 13d05ebdca8c4f489c597ce550dd773d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } }
I20240501 07:59:09.945317 9990 consensus_queue.cc:1035] T 404bb81b9c404dd288d2f8160a4273da P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:59:09.945345 9618 consensus_queue.cc:1035] T 3b9b71d44f754353b2321e8c36dd4dfd P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240501 07:59:09.934551 6644 consensus_peers.cc:485] T 7d51b1940b594c79a7d13d7026b76da8 P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131:34471): Couldn't send request to peer 3b164cbbc00f4e8eb3254ddd349a3ff9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7d51b1940b594c79a7d13d7026b76da8. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:09.946394 8793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "239ee596e62d47169eeb6670dc62d59c" candidate_uuid: "13d05ebdca8c4f489c597ce550dd773d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74077b61d6f54342bf9736b4a9f2eade"
I20240501 07:59:09.946529 8793 raft_consensus.cc:3037] T 239ee596e62d47169eeb6670dc62d59c P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:09.946892 10310 raft_consensus.cc:2258] T 7d51b1940b594c79a7d13d7026b76da8 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:09.947484 10310 ts_tablet_manager.cc:1875] T 7d51b1940b594c79a7d13d7026b76da8 P 1f9cb72be98941b5a8e504cbe34132b9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:59:09.947496 8793 raft_consensus.cc:2454] T 239ee596e62d47169eeb6670dc62d59c P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13d05ebdca8c4f489c597ce550dd773d in term 1.
I20240501 07:59:09.948071 6576 raft_consensus.cc:1261] T 3b9b71d44f754353b2321e8c36dd4dfd P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Refusing update from remote peer 1f9cb72be98941b5a8e504cbe34132b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:59:09.948071 6575 raft_consensus.cc:1261] T 404bb81b9c404dd288d2f8160a4273da P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Refusing update from remote peer 1f9cb72be98941b5a8e504cbe34132b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:59:09.949882 6170 catalog_manager.cc:5484] T 239ee596e62d47169eeb6670dc62d59c P 13d05ebdca8c4f489c597ce550dd773d reported cstate change: term changed from 0 to 1, leader changed from <none> to 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134). New cstate: current_term: 1 leader_uuid: "13d05ebdca8c4f489c597ce550dd773d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } health_report { overall_health: HEALTHY } } }
I20240501 07:59:09.950362 10288 consensus_queue.cc:1035] T 404bb81b9c404dd288d2f8160a4273da P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:59:09.950693 10288 consensus_queue.cc:1035] T 3b9b71d44f754353b2321e8c36dd4dfd P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:59:09.951354 10310 ts_tablet_manager.cc:1888] T 7d51b1940b594c79a7d13d7026b76da8 P 1f9cb72be98941b5a8e504cbe34132b9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:59:09.951432 10310 log.cc:1196] T 7d51b1940b594c79a7d13d7026b76da8 P 1f9cb72be98941b5a8e504cbe34132b9: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/7d51b1940b594c79a7d13d7026b76da8
I20240501 07:59:09.951438 10288 raft_consensus.cc:1050] T c3a9036ccfee45498bca9681518d3699 P 1f9cb72be98941b5a8e504cbe34132b9: attempting to promote NON_VOTER 74077b61d6f54342bf9736b4a9f2eade to VOTER
I20240501 07:59:09.951716 10310 ts_tablet_manager.cc:1909] T 7d51b1940b594c79a7d13d7026b76da8 P 1f9cb72be98941b5a8e504cbe34132b9: Deleting consensus metadata
I20240501 07:59:09.951882 10288 consensus_queue.cc:237] T c3a9036ccfee45498bca9681518d3699 P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { replace: true } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } }
I20240501 07:59:09.952311 10310 tablet_replica.cc:330] T 5ee738571af740c1b494d4f17413d1d8 P 1f9cb72be98941b5a8e504cbe34132b9: stopping tablet replica
I20240501 07:59:09.952437 10310 raft_consensus.cc:2229] T 5ee738571af740c1b494d4f17413d1d8 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:09.952581 10310 raft_consensus.cc:2258] T 5ee738571af740c1b494d4f17413d1d8 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:09.952791 6159 catalog_manager.cc:4830] TS 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132:36773): tablet 7d51b1940b594c79a7d13d7026b76da8 (table rebalance_test_table_8 [id=85a512f46553493eaf7ce4a1eecac55f]) successfully deleted
I20240501 07:59:09.953346 8793 raft_consensus.cc:1261] T c3a9036ccfee45498bca9681518d3699 P 74077b61d6f54342bf9736b4a9f2eade [term 1 LEARNER]: Refusing update from remote peer 1f9cb72be98941b5a8e504cbe34132b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:59:09.953352 6577 raft_consensus.cc:1261] T c3a9036ccfee45498bca9681518d3699 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Refusing update from remote peer 1f9cb72be98941b5a8e504cbe34132b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:59:09.953856 9990 consensus_queue.cc:1035] T c3a9036ccfee45498bca9681518d3699 P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:59:09.954190 9056 raft_consensus.cc:1261] T 3b9b71d44f754353b2321e8c36dd4dfd P a505e7cb133c4392bfda8a4a20874ae9 [term 1 LEARNER]: Refusing update from remote peer 1f9cb72be98941b5a8e504cbe34132b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:59:09.954294 9990 consensus_queue.cc:1035] T c3a9036ccfee45498bca9681518d3699 P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:59:09.954519 6445 raft_consensus.cc:1261] T c3a9036ccfee45498bca9681518d3699 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Refusing update from remote peer 1f9cb72be98941b5a8e504cbe34132b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:59:09.954735 9990 consensus_queue.cc:1035] T 3b9b71d44f754353b2321e8c36dd4dfd P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:59:09.954911 9398 consensus_queue.cc:1035] T c3a9036ccfee45498bca9681518d3699 P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:59:09.956902 9398 raft_consensus.cc:2932] T 3b9b71d44f754353b2321e8c36dd4dfd P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false } } }
I20240501 07:59:09.957783 9618 raft_consensus.cc:2932] T 404bb81b9c404dd288d2f8160a4273da P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false } } }
I20240501 07:59:09.958223 6575 raft_consensus.cc:2932] T 404bb81b9c404dd288d2f8160a4273da P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false } } }
I20240501 07:59:09.959215 6577 raft_consensus.cc:2932] T 3b9b71d44f754353b2321e8c36dd4dfd P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false } } }
I20240501 07:59:09.960812 10288 raft_consensus.cc:2932] T c3a9036ccfee45498bca9681518d3699 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { replace: true } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } } }
I20240501 07:59:09.961594 10309 ts_tablet_manager.cc:1888] T 7d51b1940b594c79a7d13d7026b76da8 P 8b8b3ad3aa8f4053b2167dd550f2086f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:59:09.961678 10309 log.cc:1196] T 7d51b1940b594c79a7d13d7026b76da8 P 8b8b3ad3aa8f4053b2167dd550f2086f: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/7d51b1940b594c79a7d13d7026b76da8
I20240501 07:59:09.962488 6171 catalog_manager.cc:5484] T 3b9b71d44f754353b2321e8c36dd4dfd P 1f9cb72be98941b5a8e504cbe34132b9 reported cstate change: config changed from index 4 to 5, a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1f9cb72be98941b5a8e504cbe34132b9" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 07:59:09.962673 8793 raft_consensus.cc:2932] T c3a9036ccfee45498bca9681518d3699 P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { replace: true } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } } }
I20240501 07:59:09.962956 6170 catalog_manager.cc:5484] T 404bb81b9c404dd288d2f8160a4273da P 3b164cbbc00f4e8eb3254ddd349a3ff9 reported cstate change: config changed from index 4 to 5, 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1f9cb72be98941b5a8e504cbe34132b9" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false } } }
I20240501 07:59:09.963726 6577 raft_consensus.cc:2932] T c3a9036ccfee45498bca9681518d3699 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { replace: true } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } } }
I20240501 07:59:09.963692 9056 raft_consensus.cc:2932] T 3b9b71d44f754353b2321e8c36dd4dfd P a505e7cb133c4392bfda8a4a20874ae9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false } } }
I20240501 07:59:09.965777 10309 ts_tablet_manager.cc:1909] T 7d51b1940b594c79a7d13d7026b76da8 P 8b8b3ad3aa8f4053b2167dd550f2086f: Deleting consensus metadata
I20240501 07:59:09.965719 6169 catalog_manager.cc:5484] T c3a9036ccfee45498bca9681518d3699 P 74077b61d6f54342bf9736b4a9f2eade reported cstate change: config changed from index 4 to 5, 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1f9cb72be98941b5a8e504cbe34132b9" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { replace: true } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } } }
I20240501 07:59:09.966760 10309 tablet_replica.cc:330] T 0cf117c67b524add96c9e2d887dc9e4d P 8b8b3ad3aa8f4053b2167dd550f2086f: stopping tablet replica
I20240501 07:59:09.966904 10309 raft_consensus.cc:2229] T 0cf117c67b524add96c9e2d887dc9e4d P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:09.966959 6157 catalog_manager.cc:4830] TS 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129:46875): tablet 7d51b1940b594c79a7d13d7026b76da8 (table rebalance_test_table_8 [id=85a512f46553493eaf7ce4a1eecac55f]) successfully deleted
I20240501 07:59:09.967038 10309 raft_consensus.cc:2258] T 0cf117c67b524add96c9e2d887dc9e4d P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:09.968984 6707 consensus_queue.cc:237] T 3b9b71d44f754353b2321e8c36dd4dfd P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false } }
I20240501 07:59:09.969569 10309 ts_tablet_manager.cc:1875] T 0cf117c67b524add96c9e2d887dc9e4d P 8b8b3ad3aa8f4053b2167dd550f2086f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:59:09.969566 9188 raft_consensus.cc:2932] T 404bb81b9c404dd288d2f8160a4273da P 83f20e84af4c40d09a4804b6b26e697f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false } } }
I20240501 07:59:09.969863 10310 ts_tablet_manager.cc:1875] T 5ee738571af740c1b494d4f17413d1d8 P 1f9cb72be98941b5a8e504cbe34132b9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:59:09.970809 9056 raft_consensus.cc:1261] T 3b9b71d44f754353b2321e8c36dd4dfd P a505e7cb133c4392bfda8a4a20874ae9 [term 1 FOLLOWER]: Refusing update from remote peer 1f9cb72be98941b5a8e504cbe34132b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 07:59:09.971204 9618 consensus_queue.cc:1035] T 3b9b71d44f754353b2321e8c36dd4dfd P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 07:59:09.971877 6577 raft_consensus.cc:1261] T 3b9b71d44f754353b2321e8c36dd4dfd P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Refusing update from remote peer 1f9cb72be98941b5a8e504cbe34132b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 07:59:09.972270 9618 consensus_queue.cc:1035] T 3b9b71d44f754353b2321e8c36dd4dfd P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 07:59:09.973496 6444 raft_consensus.cc:2932] T c3a9036ccfee45498bca9681518d3699 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { replace: true } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } } }
I20240501 07:59:09.973510 6445 raft_consensus.cc:2932] T 404bb81b9c404dd288d2f8160a4273da P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false } } }
I20240501 07:59:09.975106 9990 raft_consensus.cc:2932] T 3b9b71d44f754353b2321e8c36dd4dfd P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false } } }
I20240501 07:59:09.976059 9258 raft_consensus.cc:482] T 51b9e79f78894a7fa240271f9a36f72b P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:09.976162 9258 raft_consensus.cc:504] T 51b9e79f78894a7fa240271f9a36f72b P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } }
I20240501 07:59:09.976473 6159 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 3b9b71d44f754353b2321e8c36dd4dfd with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 07:59:09.976584 9258 leader_election.cc:288] T 51b9e79f78894a7fa240271f9a36f72b P 83f20e84af4c40d09a4804b6b26e697f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607), 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134:40141)
I20240501 07:59:09.977380 8924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51b9e79f78894a7fa240271f9a36f72b" candidate_uuid: "83f20e84af4c40d09a4804b6b26e697f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13d05ebdca8c4f489c597ce550dd773d" is_pre_election: true
I20240501 07:59:09.977509 8924 raft_consensus.cc:2454] T 51b9e79f78894a7fa240271f9a36f72b P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83f20e84af4c40d09a4804b6b26e697f in term 0.
I20240501 07:59:09.977974 9124 leader_election.cc:302] T 51b9e79f78894a7fa240271f9a36f72b P 83f20e84af4c40d09a4804b6b26e697f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13d05ebdca8c4f489c597ce550dd773d, 83f20e84af4c40d09a4804b6b26e697f; no voters:
I20240501 07:59:09.978310 9258 raft_consensus.cc:2781] T 51b9e79f78894a7fa240271f9a36f72b P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:09.978370 9258 raft_consensus.cc:482] T 51b9e79f78894a7fa240271f9a36f72b P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:09.978413 9258 raft_consensus.cc:3037] T 51b9e79f78894a7fa240271f9a36f72b P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:09.979139 9258 raft_consensus.cc:504] T 51b9e79f78894a7fa240271f9a36f72b P 83f20e84af4c40d09a4804b6b26e697f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } }
I20240501 07:59:09.979444 9258 leader_election.cc:288] T 51b9e79f78894a7fa240271f9a36f72b P 83f20e84af4c40d09a4804b6b26e697f [CANDIDATE]: Term 1 election: Requested vote from peers a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607), 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134:40141)
I20240501 07:59:09.979627 10309 ts_tablet_manager.cc:1888] T 0cf117c67b524add96c9e2d887dc9e4d P 8b8b3ad3aa8f4053b2167dd550f2086f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 07:59:09.979707 10309 log.cc:1196] T 0cf117c67b524add96c9e2d887dc9e4d P 8b8b3ad3aa8f4053b2167dd550f2086f: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/0cf117c67b524add96c9e2d887dc9e4d
I20240501 07:59:09.979818 9055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51b9e79f78894a7fa240271f9a36f72b" candidate_uuid: "83f20e84af4c40d09a4804b6b26e697f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a505e7cb133c4392bfda8a4a20874ae9"
I20240501 07:59:09.979935 9055 raft_consensus.cc:3037] T 51b9e79f78894a7fa240271f9a36f72b P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:09.979983 10309 ts_tablet_manager.cc:1909] T 0cf117c67b524add96c9e2d887dc9e4d P 8b8b3ad3aa8f4053b2167dd550f2086f: Deleting consensus metadata
I20240501 07:59:09.979974 10310 ts_tablet_manager.cc:1888] T 5ee738571af740c1b494d4f17413d1d8 P 1f9cb72be98941b5a8e504cbe34132b9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 07:59:09.980037 10310 log.cc:1196] T 5ee738571af740c1b494d4f17413d1d8 P 1f9cb72be98941b5a8e504cbe34132b9: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/5ee738571af740c1b494d4f17413d1d8
I20240501 07:59:09.980530 6159 catalog_manager.cc:4830] TS 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132:36773): tablet 5ee738571af740c1b494d4f17413d1d8 (table rebalance_test_table_3 [id=eb6765f7643442ff82fde7f790b2aaed]) successfully deleted
I20240501 07:59:09.980803 9055 raft_consensus.cc:2454] T 51b9e79f78894a7fa240271f9a36f72b P a505e7cb133c4392bfda8a4a20874ae9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83f20e84af4c40d09a4804b6b26e697f in term 1.
I20240501 07:59:09.981065 9123 leader_election.cc:302] T 51b9e79f78894a7fa240271f9a36f72b P 83f20e84af4c40d09a4804b6b26e697f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83f20e84af4c40d09a4804b6b26e697f, a505e7cb133c4392bfda8a4a20874ae9; no voters:
I20240501 07:59:09.981343 8924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51b9e79f78894a7fa240271f9a36f72b" candidate_uuid: "83f20e84af4c40d09a4804b6b26e697f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13d05ebdca8c4f489c597ce550dd773d"
I20240501 07:59:09.981428 9258 raft_consensus.cc:2781] T 51b9e79f78894a7fa240271f9a36f72b P 83f20e84af4c40d09a4804b6b26e697f [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:09.981467 8924 raft_consensus.cc:3037] T 51b9e79f78894a7fa240271f9a36f72b P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:09.981499 9258 raft_consensus.cc:686] T 51b9e79f78894a7fa240271f9a36f72b P 83f20e84af4c40d09a4804b6b26e697f [term 1 LEADER]: Becoming Leader. State: Replica: 83f20e84af4c40d09a4804b6b26e697f, State: Running, Role: LEADER
I20240501 07:59:09.981611 9258 consensus_queue.cc:237] T 51b9e79f78894a7fa240271f9a36f72b P 83f20e84af4c40d09a4804b6b26e697f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } }
I20240501 07:59:09.981544 9055 raft_consensus.cc:2932] T 3b9b71d44f754353b2321e8c36dd4dfd P a505e7cb133c4392bfda8a4a20874ae9 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false } } }
I20240501 07:59:09.982415 8924 raft_consensus.cc:2454] T 51b9e79f78894a7fa240271f9a36f72b P 13d05ebdca8c4f489c597ce550dd773d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83f20e84af4c40d09a4804b6b26e697f in term 1.
I20240501 07:59:09.982445 10309 tablet_replica.cc:330] T 187940ba79f546b1ad79e7ec1de76442 P 8b8b3ad3aa8f4053b2167dd550f2086f: stopping tablet replica
I20240501 07:59:09.982589 10309 raft_consensus.cc:2229] T 187940ba79f546b1ad79e7ec1de76442 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:09.983181 10309 raft_consensus.cc:2258] T 187940ba79f546b1ad79e7ec1de76442 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:09.983418 6157 catalog_manager.cc:4830] TS 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129:46875): tablet 0cf117c67b524add96c9e2d887dc9e4d (table rebalance_test_table_8 [id=85a512f46553493eaf7ce4a1eecac55f]) successfully deleted
I20240501 07:59:09.983917 6170 catalog_manager.cc:5484] T 51b9e79f78894a7fa240271f9a36f72b P 83f20e84af4c40d09a4804b6b26e697f reported cstate change: term changed from 0 to 1, leader changed from <none> to 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136). New cstate: current_term: 1 leader_uuid: "83f20e84af4c40d09a4804b6b26e697f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } health_report { overall_health: UNKNOWN } } }
I20240501 07:59:09.984113 10309 ts_tablet_manager.cc:1875] T 187940ba79f546b1ad79e7ec1de76442 P 8b8b3ad3aa8f4053b2167dd550f2086f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:59:09.983966 6166 catalog_manager.cc:5484] T 3b9b71d44f754353b2321e8c36dd4dfd P a505e7cb133c4392bfda8a4a20874ae9 reported cstate change: config changed from index 5 to 6, VOTER 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "1f9cb72be98941b5a8e504cbe34132b9" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false } } }
I20240501 07:59:09.986930 10309 ts_tablet_manager.cc:1888] T 187940ba79f546b1ad79e7ec1de76442 P 8b8b3ad3aa8f4053b2167dd550f2086f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 07:59:09.987030 10309 log.cc:1196] T 187940ba79f546b1ad79e7ec1de76442 P 8b8b3ad3aa8f4053b2167dd550f2086f: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/187940ba79f546b1ad79e7ec1de76442
I20240501 07:59:09.987018 6294 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b9b71d44f754353b2321e8c36dd4dfd with delete_type TABLET_DATA_TOMBSTONED (TS 8b8b3ad3aa8f4053b2167dd550f2086f not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:45480
I20240501 07:59:09.987512 10309 tablet_replica.cc:330] T 3b9b71d44f754353b2321e8c36dd4dfd P 8b8b3ad3aa8f4053b2167dd550f2086f: stopping tablet replica
I20240501 07:59:09.987653 10309 raft_consensus.cc:2229] T 3b9b71d44f754353b2321e8c36dd4dfd P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:09.987690 6157 catalog_manager.cc:4830] TS 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129:46875): tablet 187940ba79f546b1ad79e7ec1de76442 (table rebalance_test_table_10 [id=7b24d2d13db34edb849642d7d3eef300]) successfully deleted
I20240501 07:59:09.987813 10309 pending_rounds.cc:62] T 3b9b71d44f754353b2321e8c36dd4dfd P 8b8b3ad3aa8f4053b2167dd550f2086f: Trying to abort 1 pending ops.
I20240501 07:59:09.987903 10309 pending_rounds.cc:68] T 3b9b71d44f754353b2321e8c36dd4dfd P 8b8b3ad3aa8f4053b2167dd550f2086f: Aborting op as it isn't in flight: id { term: 1 index: 5 } timestamp: 7022798233365504000 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "3b9b71d44f754353b2321e8c36dd4dfd" old_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: true } } } new_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false } } } }
I20240501 07:59:09.988073 10309 raft_consensus.cc:2894] T 3b9b71d44f754353b2321e8c36dd4dfd P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Aborting config change with OpId 1.5: Aborted: Op aborted
I20240501 07:59:09.988162 10309 raft_consensus.cc:2866] T 3b9b71d44f754353b2321e8c36dd4dfd P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240501 07:59:09.988221 10309 raft_consensus.cc:2258] T 3b9b71d44f754353b2321e8c36dd4dfd P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:09.989035 10309 ts_tablet_manager.cc:1875] T 3b9b71d44f754353b2321e8c36dd4dfd P 8b8b3ad3aa8f4053b2167dd550f2086f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:59:09.989415 9056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51b9e79f78894a7fa240271f9a36f72b" candidate_uuid: "83f20e84af4c40d09a4804b6b26e697f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a505e7cb133c4392bfda8a4a20874ae9" is_pre_election: true
I20240501 07:59:09.989573 9056 raft_consensus.cc:2362] T 51b9e79f78894a7fa240271f9a36f72b P a505e7cb133c4392bfda8a4a20874ae9 [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 83f20e84af4c40d09a4804b6b26e697f in term 1. Re-sending same reply.
I20240501 07:59:09.989835 6707 consensus_queue.cc:237] T 404bb81b9c404dd288d2f8160a4273da P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false } }
I20240501 07:59:09.989967 6577 raft_consensus.cc:2932] T 3b9b71d44f754353b2321e8c36dd4dfd P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false } } }
I20240501 07:59:09.991251 10309 ts_tablet_manager.cc:1888] T 3b9b71d44f754353b2321e8c36dd4dfd P 8b8b3ad3aa8f4053b2167dd550f2086f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 07:59:09.991364 10309 log.cc:1196] T 3b9b71d44f754353b2321e8c36dd4dfd P 8b8b3ad3aa8f4053b2167dd550f2086f: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/3b9b71d44f754353b2321e8c36dd4dfd
I20240501 07:59:09.991717 6576 raft_consensus.cc:1261] T 404bb81b9c404dd288d2f8160a4273da P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Refusing update from remote peer 1f9cb72be98941b5a8e504cbe34132b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 07:59:09.992211 9990 consensus_queue.cc:1035] T 404bb81b9c404dd288d2f8160a4273da P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 07:59:09.992408 9188 raft_consensus.cc:1261] T 404bb81b9c404dd288d2f8160a4273da P 83f20e84af4c40d09a4804b6b26e697f [term 1 FOLLOWER]: Refusing update from remote peer 1f9cb72be98941b5a8e504cbe34132b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 07:59:09.992754 6157 catalog_manager.cc:4830] TS 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129:46875): tablet 3b9b71d44f754353b2321e8c36dd4dfd (table rebalance_test_table_15 [id=ecf5cb7235314f20bbb522737ba7fd3c]) successfully deleted
I20240501 07:59:09.992915 9618 consensus_queue.cc:1035] T 404bb81b9c404dd288d2f8160a4273da P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 07:59:09.995802 9618 raft_consensus.cc:2932] T 404bb81b9c404dd288d2f8160a4273da P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false } } }
I20240501 07:59:09.997155 9188 raft_consensus.cc:2932] T 404bb81b9c404dd288d2f8160a4273da P 83f20e84af4c40d09a4804b6b26e697f [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false } } }
I20240501 07:59:09.999644 6166 catalog_manager.cc:5484] T 404bb81b9c404dd288d2f8160a4273da P 83f20e84af4c40d09a4804b6b26e697f reported cstate change: config changed from index 5 to 6, VOTER 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "1f9cb72be98941b5a8e504cbe34132b9" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false } } }
I20240501 07:59:10.000335 6159 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 404bb81b9c404dd288d2f8160a4273da with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 07:59:10.002825 6577 raft_consensus.cc:2932] T 404bb81b9c404dd288d2f8160a4273da P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false } } }
I20240501 07:59:10.003887 6425 tablet_service.cc:1514] Processing DeleteTablet for tablet 404bb81b9c404dd288d2f8160a4273da with delete_type TABLET_DATA_TOMBSTONED (TS 8315d22c56f7439a9e1fc82f153f018c not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:45034
I20240501 07:59:10.004145 10075 tablet_replica.cc:330] T 404bb81b9c404dd288d2f8160a4273da P 8315d22c56f7439a9e1fc82f153f018c: stopping tablet replica
I20240501 07:59:10.004323 10075 raft_consensus.cc:2229] T 404bb81b9c404dd288d2f8160a4273da P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:10.004490 10075 raft_consensus.cc:2258] T 404bb81b9c404dd288d2f8160a4273da P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:10.005364 10075 ts_tablet_manager.cc:1875] T 404bb81b9c404dd288d2f8160a4273da P 8315d22c56f7439a9e1fc82f153f018c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:59:10.007426 10075 ts_tablet_manager.cc:1888] T 404bb81b9c404dd288d2f8160a4273da P 8315d22c56f7439a9e1fc82f153f018c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 07:59:10.007484 10075 log.cc:1196] T 404bb81b9c404dd288d2f8160a4273da P 8315d22c56f7439a9e1fc82f153f018c: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/404bb81b9c404dd288d2f8160a4273da
I20240501 07:59:10.007992 6159 catalog_manager.cc:4830] TS 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130:32941): tablet 404bb81b9c404dd288d2f8160a4273da (table rebalance_test_table_19 [id=35f40c7e6b1e4d799576a7d98130d848]) successfully deleted
I20240501 07:59:10.012766 8905 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:59:10.016974 6687 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 07:59:10.023783 6555 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 07:59:10.025596 10352 ts_tablet_manager.cc:902] T 3745acee2b5543b697a3c8f45d2618bb P a505e7cb133c4392bfda8a4a20874ae9: Initiating tablet copy from peer 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131:34471)
I20240501 07:59:10.026173 10352 tablet_copy_client.cc:308] T 3745acee2b5543b697a3c8f45d2618bb P a505e7cb133c4392bfda8a4a20874ae9: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:34471
I20240501 07:59:10.041666 9167 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 07:59:10.045253 6294 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 07:59:10.051090 9036 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:59:10.062986 9673 raft_consensus.cc:482] T 2976d6353e8947b7b6c37f72efbeeac4 P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:10.063109 9673 raft_consensus.cc:504] T 2976d6353e8947b7b6c37f72efbeeac4 P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } }
I20240501 07:59:10.063594 9673 leader_election.cc:288] T 2976d6353e8947b7b6c37f72efbeeac4 P 83f20e84af4c40d09a4804b6b26e697f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129:46875), a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607)
I20240501 07:59:10.064296 6597 tablet_copy_service.cc:140] P 3b164cbbc00f4e8eb3254ddd349a3ff9: Received BeginTabletCopySession request for tablet 3745acee2b5543b697a3c8f45d2618bb from peer a505e7cb133c4392bfda8a4a20874ae9 ({username='slave'} at 127.0.99.135:40821)
I20240501 07:59:10.064421 6597 tablet_copy_service.cc:161] P 3b164cbbc00f4e8eb3254ddd349a3ff9: Beginning new tablet copy session on tablet 3745acee2b5543b697a3c8f45d2618bb from peer a505e7cb133c4392bfda8a4a20874ae9 at {username='slave'} at 127.0.99.135:40821: session id = a505e7cb133c4392bfda8a4a20874ae9-3745acee2b5543b697a3c8f45d2618bb
I20240501 07:59:10.064684 9055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2976d6353e8947b7b6c37f72efbeeac4" candidate_uuid: "83f20e84af4c40d09a4804b6b26e697f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a505e7cb133c4392bfda8a4a20874ae9" is_pre_election: true
I20240501 07:59:10.064812 9055 raft_consensus.cc:2454] T 2976d6353e8947b7b6c37f72efbeeac4 P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83f20e84af4c40d09a4804b6b26e697f in term 0.
I20240501 07:59:10.065330 9123 leader_election.cc:302] T 2976d6353e8947b7b6c37f72efbeeac4 P 83f20e84af4c40d09a4804b6b26e697f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83f20e84af4c40d09a4804b6b26e697f, a505e7cb133c4392bfda8a4a20874ae9; no voters:
I20240501 07:59:10.065515 6597 tablet_copy_source_session.cc:205] T 3745acee2b5543b697a3c8f45d2618bb P 3b164cbbc00f4e8eb3254ddd349a3ff9: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:10.066449 10352 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3745acee2b5543b697a3c8f45d2618bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:10.068585 10352 tablet_copy_client.cc:783] T 3745acee2b5543b697a3c8f45d2618bb P a505e7cb133c4392bfda8a4a20874ae9: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:10.068733 10352 tablet_copy_client.cc:647] T 3745acee2b5543b697a3c8f45d2618bb P a505e7cb133c4392bfda8a4a20874ae9: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:10.068951 6425 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 07:59:10.073311 9673 raft_consensus.cc:2781] T 2976d6353e8947b7b6c37f72efbeeac4 P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:10.073401 9673 raft_consensus.cc:482] T 2976d6353e8947b7b6c37f72efbeeac4 P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:10.073453 9673 raft_consensus.cc:3037] T 2976d6353e8947b7b6c37f72efbeeac4 P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:10.074414 9673 raft_consensus.cc:504] T 2976d6353e8947b7b6c37f72efbeeac4 P 83f20e84af4c40d09a4804b6b26e697f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } }
I20240501 07:59:10.074844 9673 leader_election.cc:288] T 2976d6353e8947b7b6c37f72efbeeac4 P 83f20e84af4c40d09a4804b6b26e697f [CANDIDATE]: Term 1 election: Requested vote from peers 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129:46875), a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607)
I20240501 07:59:10.075357 9056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2976d6353e8947b7b6c37f72efbeeac4" candidate_uuid: "83f20e84af4c40d09a4804b6b26e697f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a505e7cb133c4392bfda8a4a20874ae9"
I20240501 07:59:10.075488 9056 raft_consensus.cc:3037] T 2976d6353e8947b7b6c37f72efbeeac4 P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:10.077714 8680 raft_consensus.cc:1050] T b7d57b7fd5234f50afbc859bc997f3e1 P 3b164cbbc00f4e8eb3254ddd349a3ff9: attempting to promote NON_VOTER 74077b61d6f54342bf9736b4a9f2eade to VOTER
I20240501 07:59:10.078087 8680 consensus_queue.cc:237] T b7d57b7fd5234f50afbc859bc997f3e1 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { replace: true } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } }
I20240501 07:59:10.080718 8793 raft_consensus.cc:1261] T b7d57b7fd5234f50afbc859bc997f3e1 P 74077b61d6f54342bf9736b4a9f2eade [term 1 LEARNER]: Refusing update from remote peer 3b164cbbc00f4e8eb3254ddd349a3ff9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:59:10.080901 6314 raft_consensus.cc:1261] T b7d57b7fd5234f50afbc859bc997f3e1 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Refusing update from remote peer 3b164cbbc00f4e8eb3254ddd349a3ff9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:59:10.081987 9507 consensus_queue.cc:1035] T b7d57b7fd5234f50afbc859bc997f3e1 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:59:10.082096 6707 raft_consensus.cc:1261] T b7d57b7fd5234f50afbc859bc997f3e1 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Refusing update from remote peer 3b164cbbc00f4e8eb3254ddd349a3ff9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:59:10.082455 9507 consensus_queue.cc:1035] T b7d57b7fd5234f50afbc859bc997f3e1 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:59:10.085180 8680 consensus_queue.cc:1035] T b7d57b7fd5234f50afbc859bc997f3e1 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:59:10.085834 10352 tablet_copy_client.cc:523] T 3745acee2b5543b697a3c8f45d2618bb P a505e7cb133c4392bfda8a4a20874ae9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:10.087388 9507 raft_consensus.cc:2932] T b7d57b7fd5234f50afbc859bc997f3e1 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { replace: true } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } } }
I20240501 07:59:10.088042 6707 raft_consensus.cc:2932] T b7d57b7fd5234f50afbc859bc997f3e1 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { replace: true } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } } }
I20240501 07:59:10.090126 9056 raft_consensus.cc:2454] T 2976d6353e8947b7b6c37f72efbeeac4 P a505e7cb133c4392bfda8a4a20874ae9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83f20e84af4c40d09a4804b6b26e697f in term 1.
I20240501 07:59:10.090373 10352 tablet_bootstrap.cc:492] T 3745acee2b5543b697a3c8f45d2618bb P a505e7cb133c4392bfda8a4a20874ae9: Bootstrap starting.
I20240501 07:59:10.092223 6171 catalog_manager.cc:5484] T b7d57b7fd5234f50afbc859bc997f3e1 P 1f9cb72be98941b5a8e504cbe34132b9 reported cstate change: config changed from index 4 to 5, 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { replace: true } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } } }
I20240501 07:59:10.093488 8793 raft_consensus.cc:2932] T b7d57b7fd5234f50afbc859bc997f3e1 P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { replace: true } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } } }
I20240501 07:59:10.094333 6313 raft_consensus.cc:2932] T b7d57b7fd5234f50afbc859bc997f3e1 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { replace: true } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } } }
I20240501 07:59:10.095491 10360 ts_tablet_manager.cc:902] T 72feac2a993141168ac9399d9361ef17 P 13d05ebdca8c4f489c597ce550dd773d: Initiating tablet copy from peer 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132:36773)
I20240501 07:59:10.096063 10360 tablet_copy_client.cc:308] T 72feac2a993141168ac9399d9361ef17 P 13d05ebdca8c4f489c597ce550dd773d: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:36773
I20240501 07:59:10.096380 9123 leader_election.cc:302] T 2976d6353e8947b7b6c37f72efbeeac4 P 83f20e84af4c40d09a4804b6b26e697f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83f20e84af4c40d09a4804b6b26e697f, a505e7cb133c4392bfda8a4a20874ae9; no voters:
I20240501 07:59:10.096597 9673 raft_consensus.cc:2781] T 2976d6353e8947b7b6c37f72efbeeac4 P 83f20e84af4c40d09a4804b6b26e697f [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:10.096673 9673 raft_consensus.cc:686] T 2976d6353e8947b7b6c37f72efbeeac4 P 83f20e84af4c40d09a4804b6b26e697f [term 1 LEADER]: Becoming Leader. State: Replica: 83f20e84af4c40d09a4804b6b26e697f, State: Running, Role: LEADER
I20240501 07:59:10.096820 9673 consensus_queue.cc:237] T 2976d6353e8947b7b6c37f72efbeeac4 P 83f20e84af4c40d09a4804b6b26e697f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } }
I20240501 07:59:10.097671 6577 consensus_queue.cc:237] T b7d57b7fd5234f50afbc859bc997f3e1 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } }
I20240501 07:59:10.098678 6728 tablet_copy_service.cc:140] P 1f9cb72be98941b5a8e504cbe34132b9: Received BeginTabletCopySession request for tablet 72feac2a993141168ac9399d9361ef17 from peer 13d05ebdca8c4f489c597ce550dd773d ({username='slave'} at 127.0.99.134:44489)
I20240501 07:59:10.098783 6728 tablet_copy_service.cc:161] P 1f9cb72be98941b5a8e504cbe34132b9: Beginning new tablet copy session on tablet 72feac2a993141168ac9399d9361ef17 from peer 13d05ebdca8c4f489c597ce550dd773d at {username='slave'} at 127.0.99.134:44489: session id = 13d05ebdca8c4f489c597ce550dd773d-72feac2a993141168ac9399d9361ef17
I20240501 07:59:10.098930 6170 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55950:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 07:59:10.099745 6728 tablet_copy_source_session.cc:205] T 72feac2a993141168ac9399d9361ef17 P 1f9cb72be98941b5a8e504cbe34132b9: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:10.101722 10360 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72feac2a993141168ac9399d9361ef17. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:10.101929 9398 raft_consensus.cc:1050] T 8454f24b0c28451b8ee51be39ca3e07f P 1f9cb72be98941b5a8e504cbe34132b9: attempting to promote NON_VOTER a505e7cb133c4392bfda8a4a20874ae9 to VOTER
I20240501 07:59:10.102342 9398 consensus_queue.cc:237] T 8454f24b0c28451b8ee51be39ca3e07f P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false } }
I20240501 07:59:10.102542 6171 catalog_manager.cc:5484] T 2976d6353e8947b7b6c37f72efbeeac4 P 83f20e84af4c40d09a4804b6b26e697f reported cstate change: term changed from 0 to 1, leader changed from <none> to 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136). New cstate: current_term: 1 leader_uuid: "83f20e84af4c40d09a4804b6b26e697f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } health_report { overall_health: HEALTHY } } }
I20240501 07:59:10.103991 8924 raft_consensus.cc:1261] T 51b9e79f78894a7fa240271f9a36f72b P 13d05ebdca8c4f489c597ce550dd773d [term 1 FOLLOWER]: Refusing update from remote peer 83f20e84af4c40d09a4804b6b26e697f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:10.104401 9055 raft_consensus.cc:1261] T 51b9e79f78894a7fa240271f9a36f72b P a505e7cb133c4392bfda8a4a20874ae9 [term 1 FOLLOWER]: Refusing update from remote peer 83f20e84af4c40d09a4804b6b26e697f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:10.105115 6314 raft_consensus.cc:1261] T 8454f24b0c28451b8ee51be39ca3e07f P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Refusing update from remote peer 1f9cb72be98941b5a8e504cbe34132b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:59:10.105434 9673 consensus_queue.cc:1035] T 51b9e79f78894a7fa240271f9a36f72b P 83f20e84af4c40d09a4804b6b26e697f [LEADER]: Connected to new peer: Peer: permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:10.105604 10196 consensus_queue.cc:1035] T 51b9e79f78894a7fa240271f9a36f72b P 83f20e84af4c40d09a4804b6b26e697f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:10.105973 10360 tablet_copy_client.cc:783] T 72feac2a993141168ac9399d9361ef17 P 13d05ebdca8c4f489c597ce550dd773d: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:10.106127 10360 tablet_copy_client.cc:647] T 72feac2a993141168ac9399d9361ef17 P 13d05ebdca8c4f489c597ce550dd773d: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:10.106654 9398 consensus_queue.cc:1035] T 8454f24b0c28451b8ee51be39ca3e07f P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:59:10.106966 6313 raft_consensus.cc:1261] T b7d57b7fd5234f50afbc859bc997f3e1 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Refusing update from remote peer 3b164cbbc00f4e8eb3254ddd349a3ff9: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 07:59:10.107448 10046 consensus_queue.cc:1035] T b7d57b7fd5234f50afbc859bc997f3e1 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 07:59:10.107671 6169 catalog_manager.cc:5576] TS a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607) does not have the latest schema for tablet 51b9e79f78894a7fa240271f9a36f72b (table rebalancer_dynamic_table [id=4fa2f76148be476390669c4d79dbb6bd]). Expected version 1 got 0
I20240501 07:59:10.108263 9056 raft_consensus.cc:1261] T 8454f24b0c28451b8ee51be39ca3e07f P a505e7cb133c4392bfda8a4a20874ae9 [term 1 LEARNER]: Refusing update from remote peer 1f9cb72be98941b5a8e504cbe34132b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:59:10.108417 6168 catalog_manager.cc:5576] TS 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134:40141) does not have the latest schema for tablet 51b9e79f78894a7fa240271f9a36f72b (table rebalancer_dynamic_table [id=4fa2f76148be476390669c4d79dbb6bd]). Expected version 1 got 0
I20240501 07:59:10.108857 8793 raft_consensus.cc:1261] T b7d57b7fd5234f50afbc859bc997f3e1 P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Refusing update from remote peer 3b164cbbc00f4e8eb3254ddd349a3ff9: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 07:59:10.109180 9398 consensus_queue.cc:1035] T 8454f24b0c28451b8ee51be39ca3e07f P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:59:10.109680 10237 consensus_queue.cc:1035] T b7d57b7fd5234f50afbc859bc997f3e1 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 07:59:10.109968 10237 raft_consensus.cc:2932] T b7d57b7fd5234f50afbc859bc997f3e1 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } } }
I20240501 07:59:10.111959 6577 raft_consensus.cc:1261] T 8454f24b0c28451b8ee51be39ca3e07f P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Refusing update from remote peer 1f9cb72be98941b5a8e504cbe34132b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:59:10.112354 9398 consensus_queue.cc:1035] T 8454f24b0c28451b8ee51be39ca3e07f P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:59:10.112388 6160 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b7d57b7fd5234f50afbc859bc997f3e1 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 07:59:10.113883 6314 raft_consensus.cc:2932] T b7d57b7fd5234f50afbc859bc997f3e1 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } } }
I20240501 07:59:10.114151 8793 raft_consensus.cc:2932] T b7d57b7fd5234f50afbc859bc997f3e1 P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } } }
I20240501 07:59:10.116753 6168 catalog_manager.cc:5484] T b7d57b7fd5234f50afbc859bc997f3e1 P 74077b61d6f54342bf9736b4a9f2eade reported cstate change: config changed from index 5 to 6, VOTER 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } } }
I20240501 07:59:10.117658 6170 catalog_manager.cc:5576] TS 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149) does not have the latest schema for tablet 51b9e79f78894a7fa240271f9a36f72b (table rebalancer_dynamic_table [id=4fa2f76148be476390669c4d79dbb6bd]). Expected version 1 got 0
I20240501 07:59:10.117794 10373 tablet.cc:1722] T 51b9e79f78894a7fa240271f9a36f72b P 83f20e84af4c40d09a4804b6b26e697f: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 07:59:10.118199 6169 catalog_manager.cc:5576] TS a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607) does not have the latest schema for tablet 51b9e79f78894a7fa240271f9a36f72b (table rebalancer_dynamic_table [id=4fa2f76148be476390669c4d79dbb6bd]). Expected version 1 got 0
I20240501 07:59:10.118990 10373 tablet.cc:1620] T 51b9e79f78894a7fa240271f9a36f72b P 83f20e84af4c40d09a4804b6b26e697f: MemRowSet was empty: no flush needed.
I20240501 07:59:10.119544 10375 tablet.cc:1722] T 51b9e79f78894a7fa240271f9a36f72b P a505e7cb133c4392bfda8a4a20874ae9: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 07:59:10.119969 10375 tablet.cc:1620] T 51b9e79f78894a7fa240271f9a36f72b P a505e7cb133c4392bfda8a4a20874ae9: MemRowSet was empty: no flush needed.
I20240501 07:59:10.120733 6687 tablet_service.cc:1514] Processing DeleteTablet for tablet b7d57b7fd5234f50afbc859bc997f3e1 with delete_type TABLET_DATA_TOMBSTONED (TS 1f9cb72be98941b5a8e504cbe34132b9 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:38524
I20240501 07:59:10.121004 10310 tablet_replica.cc:330] T b7d57b7fd5234f50afbc859bc997f3e1 P 1f9cb72be98941b5a8e504cbe34132b9: stopping tablet replica
I20240501 07:59:10.121158 10310 raft_consensus.cc:2229] T b7d57b7fd5234f50afbc859bc997f3e1 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:10.121341 10310 raft_consensus.cc:2258] T b7d57b7fd5234f50afbc859bc997f3e1 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:10.126451 6314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2976d6353e8947b7b6c37f72efbeeac4" candidate_uuid: "83f20e84af4c40d09a4804b6b26e697f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" is_pre_election: true
I20240501 07:59:10.126611 6314 raft_consensus.cc:2454] T 2976d6353e8947b7b6c37f72efbeeac4 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83f20e84af4c40d09a4804b6b26e697f in term 0.
I20240501 07:59:10.126466 6311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2976d6353e8947b7b6c37f72efbeeac4" candidate_uuid: "83f20e84af4c40d09a4804b6b26e697f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f"
I20240501 07:59:10.126981 6311 raft_consensus.cc:3037] T 2976d6353e8947b7b6c37f72efbeeac4 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:10.127499 10324 raft_consensus.cc:2932] T 8454f24b0c28451b8ee51be39ca3e07f P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false } } }
I20240501 07:59:10.128063 6311 raft_consensus.cc:2454] T 2976d6353e8947b7b6c37f72efbeeac4 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83f20e84af4c40d09a4804b6b26e697f in term 1.
I20240501 07:59:10.129544 6171 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 0bc706e85ce74cef97a4827ea4d4aa65: rebalancer_dynamic_table [id=4fa2f76148be476390669c4d79dbb6bd] alter complete (version 1)
I20240501 07:59:10.129925 6576 raft_consensus.cc:2932] T 8454f24b0c28451b8ee51be39ca3e07f P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false } } }
I20240501 07:59:10.130038 9055 raft_consensus.cc:2932] T 8454f24b0c28451b8ee51be39ca3e07f P a505e7cb133c4392bfda8a4a20874ae9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false } } }
I20240501 07:59:10.133242 6164 catalog_manager.cc:5484] T 8454f24b0c28451b8ee51be39ca3e07f P 1f9cb72be98941b5a8e504cbe34132b9 reported cstate change: config changed from index 4 to 5, a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1f9cb72be98941b5a8e504cbe34132b9" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 07:59:10.134152 10360 tablet_copy_client.cc:523] T 72feac2a993141168ac9399d9361ef17 P 13d05ebdca8c4f489c597ce550dd773d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:10.134331 6169 catalog_manager.cc:5576] TS 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134:40141) does not have the latest schema for tablet 51b9e79f78894a7fa240271f9a36f72b (table rebalancer_dynamic_table [id=4fa2f76148be476390669c4d79dbb6bd]). Expected version 1 got 0
I20240501 07:59:10.135645 10360 tablet_bootstrap.cc:492] T 72feac2a993141168ac9399d9361ef17 P 13d05ebdca8c4f489c597ce550dd773d: Bootstrap starting.
I20240501 07:59:10.136418 10379 ts_tablet_manager.cc:902] T f9df6f3df2b94e5eb97dbff04dbe2ae1 P 83f20e84af4c40d09a4804b6b26e697f: Initiating tablet copy from peer 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130:32941)
I20240501 07:59:10.137022 10379 tablet_copy_client.cc:308] T f9df6f3df2b94e5eb97dbff04dbe2ae1 P 83f20e84af4c40d09a4804b6b26e697f: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:32941
I20240501 07:59:10.137467 6466 tablet_copy_service.cc:140] P 8315d22c56f7439a9e1fc82f153f018c: Received BeginTabletCopySession request for tablet f9df6f3df2b94e5eb97dbff04dbe2ae1 from peer 83f20e84af4c40d09a4804b6b26e697f ({username='slave'} at 127.0.99.136:51357)
I20240501 07:59:10.137596 6466 tablet_copy_service.cc:161] P 8315d22c56f7439a9e1fc82f153f018c: Beginning new tablet copy session on tablet f9df6f3df2b94e5eb97dbff04dbe2ae1 from peer 83f20e84af4c40d09a4804b6b26e697f at {username='slave'} at 127.0.99.136:51357: session id = 83f20e84af4c40d09a4804b6b26e697f-f9df6f3df2b94e5eb97dbff04dbe2ae1
I20240501 07:59:10.138402 6466 tablet_copy_source_session.cc:205] T f9df6f3df2b94e5eb97dbff04dbe2ae1 P 8315d22c56f7439a9e1fc82f153f018c: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:10.139212 10379 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9df6f3df2b94e5eb97dbff04dbe2ae1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:10.139938 10310 ts_tablet_manager.cc:1875] T b7d57b7fd5234f50afbc859bc997f3e1 P 1f9cb72be98941b5a8e504cbe34132b9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:59:10.140755 6313 raft_consensus.cc:2932] T 8454f24b0c28451b8ee51be39ca3e07f P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false } } }
I20240501 07:59:10.141954 10378 tablet.cc:1722] T 51b9e79f78894a7fa240271f9a36f72b P 13d05ebdca8c4f489c597ce550dd773d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 07:59:10.142167 10310 ts_tablet_manager.cc:1888] T b7d57b7fd5234f50afbc859bc997f3e1 P 1f9cb72be98941b5a8e504cbe34132b9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 07:59:10.142318 10310 log.cc:1196] T b7d57b7fd5234f50afbc859bc997f3e1 P 1f9cb72be98941b5a8e504cbe34132b9: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/b7d57b7fd5234f50afbc859bc997f3e1
I20240501 07:59:10.142693 10378 tablet.cc:1620] T 51b9e79f78894a7fa240271f9a36f72b P 13d05ebdca8c4f489c597ce550dd773d: MemRowSet was empty: no flush needed.
I20240501 07:59:10.142947 6159 catalog_manager.cc:4830] TS 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132:36773): tablet b7d57b7fd5234f50afbc859bc997f3e1 (table rebalance_test_table_11 [id=3e062fd8d58941efbd3d190c28bb0059]) successfully deleted
I20240501 07:59:10.143097 8757 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 07:59:10.145674 6707 consensus_queue.cc:237] T 8454f24b0c28451b8ee51be39ca3e07f P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false } }
I20240501 07:59:10.146536 10379 tablet_copy_client.cc:783] T f9df6f3df2b94e5eb97dbff04dbe2ae1 P 83f20e84af4c40d09a4804b6b26e697f: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:10.146690 10379 tablet_copy_client.cc:647] T f9df6f3df2b94e5eb97dbff04dbe2ae1 P 83f20e84af4c40d09a4804b6b26e697f: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:10.147362 9055 raft_consensus.cc:1261] T 8454f24b0c28451b8ee51be39ca3e07f P a505e7cb133c4392bfda8a4a20874ae9 [term 1 FOLLOWER]: Refusing update from remote peer 1f9cb72be98941b5a8e504cbe34132b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 07:59:10.148334 9618 consensus_queue.cc:1035] T 8454f24b0c28451b8ee51be39ca3e07f P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 07:59:10.148885 6313 raft_consensus.cc:1261] T 8454f24b0c28451b8ee51be39ca3e07f P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Refusing update from remote peer 1f9cb72be98941b5a8e504cbe34132b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 07:59:10.149705 9618 consensus_queue.cc:1035] T 8454f24b0c28451b8ee51be39ca3e07f P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 07:59:10.150532 9618 raft_consensus.cc:2932] T 8454f24b0c28451b8ee51be39ca3e07f P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false } } }
I20240501 07:59:10.151136 9055 raft_consensus.cc:2932] T 8454f24b0c28451b8ee51be39ca3e07f P a505e7cb133c4392bfda8a4a20874ae9 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false } } }
I20240501 07:59:10.152566 6159 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8454f24b0c28451b8ee51be39ca3e07f with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 07:59:10.153519 6168 catalog_manager.cc:5484] T 8454f24b0c28451b8ee51be39ca3e07f P 1f9cb72be98941b5a8e504cbe34132b9 reported cstate change: config changed from index 5 to 6, VOTER 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "1f9cb72be98941b5a8e504cbe34132b9" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 07:59:10.155215 6313 raft_consensus.cc:2932] T 8454f24b0c28451b8ee51be39ca3e07f P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false } } }
I20240501 07:59:10.157272 6555 tablet_service.cc:1514] Processing DeleteTablet for tablet 8454f24b0c28451b8ee51be39ca3e07f with delete_type TABLET_DATA_TOMBSTONED (TS 3b164cbbc00f4e8eb3254ddd349a3ff9 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:53874
I20240501 07:59:10.157541 9840 tablet_replica.cc:330] T 8454f24b0c28451b8ee51be39ca3e07f P 3b164cbbc00f4e8eb3254ddd349a3ff9: stopping tablet replica
I20240501 07:59:10.157699 9840 raft_consensus.cc:2229] T 8454f24b0c28451b8ee51be39ca3e07f P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:10.157879 9840 raft_consensus.cc:2258] T 8454f24b0c28451b8ee51be39ca3e07f P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:10.158607 9840 ts_tablet_manager.cc:1875] T 8454f24b0c28451b8ee51be39ca3e07f P 3b164cbbc00f4e8eb3254ddd349a3ff9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:59:10.160815 9840 ts_tablet_manager.cc:1888] T 8454f24b0c28451b8ee51be39ca3e07f P 3b164cbbc00f4e8eb3254ddd349a3ff9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 07:59:10.160893 9840 log.cc:1196] T 8454f24b0c28451b8ee51be39ca3e07f P 3b164cbbc00f4e8eb3254ddd349a3ff9: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/8454f24b0c28451b8ee51be39ca3e07f
I20240501 07:59:10.161537 6160 catalog_manager.cc:4830] TS 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131:34471): tablet 8454f24b0c28451b8ee51be39ca3e07f (table rebalance_test_table_7 [id=8e242b97203b46d0840cb3b6a649e5a7]) successfully deleted
I20240501 07:59:10.174238 10379 tablet_copy_client.cc:523] T f9df6f3df2b94e5eb97dbff04dbe2ae1 P 83f20e84af4c40d09a4804b6b26e697f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:10.175778 10379 tablet_bootstrap.cc:492] T f9df6f3df2b94e5eb97dbff04dbe2ae1 P 83f20e84af4c40d09a4804b6b26e697f: Bootstrap starting.
I20240501 07:59:10.183516 10360 tablet_bootstrap.cc:492] T 72feac2a993141168ac9399d9361ef17 P 13d05ebdca8c4f489c597ce550dd773d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:10.183934 10360 tablet_bootstrap.cc:492] T 72feac2a993141168ac9399d9361ef17 P 13d05ebdca8c4f489c597ce550dd773d: Bootstrap complete.
I20240501 07:59:10.184206 10360 ts_tablet_manager.cc:1367] T 72feac2a993141168ac9399d9361ef17 P 13d05ebdca8c4f489c597ce550dd773d: Time spent bootstrapping tablet: real 0.049s user 0.028s sys 0.000s
I20240501 07:59:10.184837 10360 raft_consensus.cc:348] T 72feac2a993141168ac9399d9361ef17 P 13d05ebdca8c4f489c597ce550dd773d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: true } }
I20240501 07:59:10.185144 10360 raft_consensus.cc:729] T 72feac2a993141168ac9399d9361ef17 P 13d05ebdca8c4f489c597ce550dd773d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 13d05ebdca8c4f489c597ce550dd773d, State: Initialized, Role: LEARNER
I20240501 07:59:10.185298 10360 consensus_queue.cc:260] T 72feac2a993141168ac9399d9361ef17 P 13d05ebdca8c4f489c597ce550dd773d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: true } }
I20240501 07:59:10.186334 10360 ts_tablet_manager.cc:1398] T 72feac2a993141168ac9399d9361ef17 P 13d05ebdca8c4f489c597ce550dd773d: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 07:59:10.187346 6728 tablet_copy_service.cc:342] P 1f9cb72be98941b5a8e504cbe34132b9: Request end of tablet copy session 13d05ebdca8c4f489c597ce550dd773d-72feac2a993141168ac9399d9361ef17 received from {username='slave'} at 127.0.99.134:44489
I20240501 07:59:10.187446 6728 tablet_copy_service.cc:433] P 1f9cb72be98941b5a8e504cbe34132b9: ending tablet copy session 13d05ebdca8c4f489c597ce550dd773d-72feac2a993141168ac9399d9361ef17 on tablet 72feac2a993141168ac9399d9361ef17 with peer 13d05ebdca8c4f489c597ce550dd773d
I20240501 07:59:10.191099 10352 tablet_bootstrap.cc:492] T 3745acee2b5543b697a3c8f45d2618bb P a505e7cb133c4392bfda8a4a20874ae9: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:10.191479 10352 tablet_bootstrap.cc:492] T 3745acee2b5543b697a3c8f45d2618bb P a505e7cb133c4392bfda8a4a20874ae9: Bootstrap complete.
I20240501 07:59:10.191696 10352 ts_tablet_manager.cc:1367] T 3745acee2b5543b697a3c8f45d2618bb P a505e7cb133c4392bfda8a4a20874ae9: Time spent bootstrapping tablet: real 0.101s user 0.028s sys 0.000s
I20240501 07:59:10.192225 10352 raft_consensus.cc:348] T 3745acee2b5543b697a3c8f45d2618bb P a505e7cb133c4392bfda8a4a20874ae9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: true } }
I20240501 07:59:10.192423 10352 raft_consensus.cc:729] T 3745acee2b5543b697a3c8f45d2618bb P a505e7cb133c4392bfda8a4a20874ae9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a505e7cb133c4392bfda8a4a20874ae9, State: Initialized, Role: LEARNER
I20240501 07:59:10.192574 10352 consensus_queue.cc:260] T 3745acee2b5543b697a3c8f45d2618bb P a505e7cb133c4392bfda8a4a20874ae9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: true } }
I20240501 07:59:10.193584 10352 ts_tablet_manager.cc:1398] T 3745acee2b5543b697a3c8f45d2618bb P a505e7cb133c4392bfda8a4a20874ae9: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:59:10.194147 6597 tablet_copy_service.cc:342] P 3b164cbbc00f4e8eb3254ddd349a3ff9: Request end of tablet copy session a505e7cb133c4392bfda8a4a20874ae9-3745acee2b5543b697a3c8f45d2618bb received from {username='slave'} at 127.0.99.135:40821
I20240501 07:59:10.194245 6597 tablet_copy_service.cc:433] P 3b164cbbc00f4e8eb3254ddd349a3ff9: ending tablet copy session a505e7cb133c4392bfda8a4a20874ae9-3745acee2b5543b697a3c8f45d2618bb on tablet 3745acee2b5543b697a3c8f45d2618bb with peer a505e7cb133c4392bfda8a4a20874ae9
I20240501 07:59:10.204073 10379 tablet_bootstrap.cc:492] T f9df6f3df2b94e5eb97dbff04dbe2ae1 P 83f20e84af4c40d09a4804b6b26e697f: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:10.204439 10379 tablet_bootstrap.cc:492] T f9df6f3df2b94e5eb97dbff04dbe2ae1 P 83f20e84af4c40d09a4804b6b26e697f: Bootstrap complete.
I20240501 07:59:10.204628 10379 ts_tablet_manager.cc:1367] T f9df6f3df2b94e5eb97dbff04dbe2ae1 P 83f20e84af4c40d09a4804b6b26e697f: Time spent bootstrapping tablet: real 0.029s user 0.026s sys 0.000s
I20240501 07:59:10.205312 10379 raft_consensus.cc:348] T f9df6f3df2b94e5eb97dbff04dbe2ae1 P 83f20e84af4c40d09a4804b6b26e697f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { replace: true } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: true } }
I20240501 07:59:10.205435 10379 raft_consensus.cc:729] T f9df6f3df2b94e5eb97dbff04dbe2ae1 P 83f20e84af4c40d09a4804b6b26e697f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 83f20e84af4c40d09a4804b6b26e697f, State: Initialized, Role: LEARNER
I20240501 07:59:10.205536 10379 consensus_queue.cc:260] T f9df6f3df2b94e5eb97dbff04dbe2ae1 P 83f20e84af4c40d09a4804b6b26e697f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { replace: true } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: true } }
I20240501 07:59:10.205801 10379 ts_tablet_manager.cc:1398] T f9df6f3df2b94e5eb97dbff04dbe2ae1 P 83f20e84af4c40d09a4804b6b26e697f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:10.206266 6466 tablet_copy_service.cc:342] P 8315d22c56f7439a9e1fc82f153f018c: Request end of tablet copy session 83f20e84af4c40d09a4804b6b26e697f-f9df6f3df2b94e5eb97dbff04dbe2ae1 received from {username='slave'} at 127.0.99.136:51357
I20240501 07:59:10.206354 6466 tablet_copy_service.cc:433] P 8315d22c56f7439a9e1fc82f153f018c: ending tablet copy session 83f20e84af4c40d09a4804b6b26e697f-f9df6f3df2b94e5eb97dbff04dbe2ae1 on tablet f9df6f3df2b94e5eb97dbff04dbe2ae1 with peer 83f20e84af4c40d09a4804b6b26e697f
I20240501 07:59:10.215790 6445 consensus_queue.cc:237] T adf207c019994dd49671bab4145fc268 P 8315d22c56f7439a9e1fc82f153f018c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { replace: true } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: true } }
I20240501 07:59:10.217262 6313 raft_consensus.cc:1261] T adf207c019994dd49671bab4145fc268 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Refusing update from remote peer 8315d22c56f7439a9e1fc82f153f018c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 07:59:10.217810 6380 consensus_peers.cc:485] T adf207c019994dd49671bab4145fc268 P 8315d22c56f7439a9e1fc82f153f018c -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: adf207c019994dd49671bab4145fc268. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:10.217833 9489 consensus_queue.cc:1035] T adf207c019994dd49671bab4145fc268 P 8315d22c56f7439a9e1fc82f153f018c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:10.218185 6707 raft_consensus.cc:1261] T adf207c019994dd49671bab4145fc268 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Refusing update from remote peer 8315d22c56f7439a9e1fc82f153f018c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:59:10.218673 9489 consensus_queue.cc:1035] T adf207c019994dd49671bab4145fc268 P 8315d22c56f7439a9e1fc82f153f018c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:10.220392 8512 raft_consensus.cc:2932] T adf207c019994dd49671bab4145fc268 P 8315d22c56f7439a9e1fc82f153f018c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { replace: true } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: true } } }
I20240501 07:59:10.221302 6313 raft_consensus.cc:2932] T adf207c019994dd49671bab4145fc268 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { replace: true } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: true } } }
I20240501 07:59:10.222725 6707 raft_consensus.cc:2932] T adf207c019994dd49671bab4145fc268 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { replace: true } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: true } } }
I20240501 07:59:10.225049 6168 catalog_manager.cc:5484] T adf207c019994dd49671bab4145fc268 P 8315d22c56f7439a9e1fc82f153f018c reported cstate change: config changed from index -1 to 4, NON_VOTER a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "8315d22c56f7439a9e1fc82f153f018c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 07:59:10.233927 10069 raft_consensus.cc:482] T 2f9e82c3f1df4d1292e37e9cea07186e P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:10.234035 10069 raft_consensus.cc:504] T 2f9e82c3f1df4d1292e37e9cea07186e P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } }
I20240501 07:59:10.234431 10069 leader_election.cc:288] T 2f9e82c3f1df4d1292e37e9cea07186e P 74077b61d6f54342bf9736b4a9f2eade [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134:40141), 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129:46875)
I20240501 07:59:10.236369 8924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f9e82c3f1df4d1292e37e9cea07186e" candidate_uuid: "74077b61d6f54342bf9736b4a9f2eade" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13d05ebdca8c4f489c597ce550dd773d" is_pre_election: true
I20240501 07:59:10.236511 8924 raft_consensus.cc:2454] T 2f9e82c3f1df4d1292e37e9cea07186e P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74077b61d6f54342bf9736b4a9f2eade in term 0.
I20240501 07:59:10.236490 6313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f9e82c3f1df4d1292e37e9cea07186e" candidate_uuid: "74077b61d6f54342bf9736b4a9f2eade" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" is_pre_election: true
I20240501 07:59:10.236646 6313 raft_consensus.cc:2454] T 2f9e82c3f1df4d1292e37e9cea07186e P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74077b61d6f54342bf9736b4a9f2eade in term 0.
I20240501 07:59:10.237043 8727 leader_election.cc:302] T 2f9e82c3f1df4d1292e37e9cea07186e P 74077b61d6f54342bf9736b4a9f2eade [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74077b61d6f54342bf9736b4a9f2eade, 8b8b3ad3aa8f4053b2167dd550f2086f; no voters:
I20240501 07:59:10.238054 6164 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55950:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 07:59:10.238627 10069 raft_consensus.cc:2781] T 2f9e82c3f1df4d1292e37e9cea07186e P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:10.238688 10069 raft_consensus.cc:482] T 2f9e82c3f1df4d1292e37e9cea07186e P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:10.238721 10069 raft_consensus.cc:3037] T 2f9e82c3f1df4d1292e37e9cea07186e P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:10.239725 10069 raft_consensus.cc:504] T 2f9e82c3f1df4d1292e37e9cea07186e P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } }
I20240501 07:59:10.240057 10069 leader_election.cc:288] T 2f9e82c3f1df4d1292e37e9cea07186e P 74077b61d6f54342bf9736b4a9f2eade [CANDIDATE]: Term 1 election: Requested vote from peers 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134:40141), 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129:46875)
I20240501 07:59:10.240388 6313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f9e82c3f1df4d1292e37e9cea07186e" candidate_uuid: "74077b61d6f54342bf9736b4a9f2eade" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f"
I20240501 07:59:10.240515 6313 raft_consensus.cc:3037] T 2f9e82c3f1df4d1292e37e9cea07186e P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:10.240881 8924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f9e82c3f1df4d1292e37e9cea07186e" candidate_uuid: "74077b61d6f54342bf9736b4a9f2eade" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13d05ebdca8c4f489c597ce550dd773d"
I20240501 07:59:10.241043 8924 raft_consensus.cc:3037] T 2f9e82c3f1df4d1292e37e9cea07186e P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:10.241578 6313 raft_consensus.cc:2454] T 2f9e82c3f1df4d1292e37e9cea07186e P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74077b61d6f54342bf9736b4a9f2eade in term 1.
I20240501 07:59:10.241930 8924 raft_consensus.cc:2454] T 2f9e82c3f1df4d1292e37e9cea07186e P 13d05ebdca8c4f489c597ce550dd773d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74077b61d6f54342bf9736b4a9f2eade in term 1.
I20240501 07:59:10.242587 8727 leader_election.cc:302] T 2f9e82c3f1df4d1292e37e9cea07186e P 74077b61d6f54342bf9736b4a9f2eade [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 13d05ebdca8c4f489c597ce550dd773d, 74077b61d6f54342bf9736b4a9f2eade, 8b8b3ad3aa8f4053b2167dd550f2086f; no voters:
I20240501 07:59:10.242748 10069 raft_consensus.cc:2781] T 2f9e82c3f1df4d1292e37e9cea07186e P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:10.242897 10069 raft_consensus.cc:686] T 2f9e82c3f1df4d1292e37e9cea07186e P 74077b61d6f54342bf9736b4a9f2eade [term 1 LEADER]: Becoming Leader. State: Replica: 74077b61d6f54342bf9736b4a9f2eade, State: Running, Role: LEADER
I20240501 07:59:10.243029 10069 consensus_queue.cc:237] T 2f9e82c3f1df4d1292e37e9cea07186e P 74077b61d6f54342bf9736b4a9f2eade [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } }
I20240501 07:59:10.245383 10373 tablet.cc:1722] T 51b9e79f78894a7fa240271f9a36f72b P 83f20e84af4c40d09a4804b6b26e697f: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 07:59:10.245702 10373 tablet.cc:1620] T 51b9e79f78894a7fa240271f9a36f72b P 83f20e84af4c40d09a4804b6b26e697f: MemRowSet was empty: no flush needed.
I20240501 07:59:10.246088 6164 catalog_manager.cc:5484] T 2f9e82c3f1df4d1292e37e9cea07186e P 74077b61d6f54342bf9736b4a9f2eade reported cstate change: term changed from 0 to 1, leader changed from <none> to 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133). New cstate: current_term: 1 leader_uuid: "74077b61d6f54342bf9736b4a9f2eade" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } health_report { overall_health: UNKNOWN } } }
I20240501 07:59:10.246759 10375 tablet.cc:1722] T 51b9e79f78894a7fa240271f9a36f72b P a505e7cb133c4392bfda8a4a20874ae9: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 07:59:10.247043 10375 tablet.cc:1620] T 51b9e79f78894a7fa240271f9a36f72b P a505e7cb133c4392bfda8a4a20874ae9: MemRowSet was empty: no flush needed.
I20240501 07:59:10.250962 10378 tablet.cc:1722] T 51b9e79f78894a7fa240271f9a36f72b P 13d05ebdca8c4f489c597ce550dd773d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 07:59:10.251261 10378 tablet.cc:1620] T 51b9e79f78894a7fa240271f9a36f72b P 13d05ebdca8c4f489c597ce550dd773d: MemRowSet was empty: no flush needed.
I20240501 07:59:10.251864 6168 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 0bc706e85ce74cef97a4827ea4d4aa65: rebalancer_dynamic_table [id=4fa2f76148be476390669c4d79dbb6bd] alter complete (version 2)
I20240501 07:59:10.295012 6576 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ff9dc384dbac47dc9795617c64688943"
dest_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:47146
I20240501 07:59:10.295185 6576 raft_consensus.cc:595] T ff9dc384dbac47dc9795617c64688943 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 LEADER]: Received request to transfer leadership
I20240501 07:59:10.303741 9507 raft_consensus.cc:979] T ff9dc384dbac47dc9795617c64688943 P 3b164cbbc00f4e8eb3254ddd349a3ff9: : Instructing follower 1f9cb72be98941b5a8e504cbe34132b9 to start an election
I20240501 07:59:10.303848 9507 raft_consensus.cc:1067] T ff9dc384dbac47dc9795617c64688943 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 LEADER]: Signalling peer 1f9cb72be98941b5a8e504cbe34132b9 to start an election
I20240501 07:59:10.304395 6707 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ff9dc384dbac47dc9795617c64688943"
dest_uuid: "1f9cb72be98941b5a8e504cbe34132b9"
from {username='slave'} at 127.0.99.131:60239
I20240501 07:59:10.304566 6707 raft_consensus.cc:482] T ff9dc384dbac47dc9795617c64688943 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:59:10.304620 6707 raft_consensus.cc:3037] T ff9dc384dbac47dc9795617c64688943 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:10.305630 6707 raft_consensus.cc:504] T ff9dc384dbac47dc9795617c64688943 P 1f9cb72be98941b5a8e504cbe34132b9 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false } }
I20240501 07:59:10.306005 6707 leader_election.cc:288] T ff9dc384dbac47dc9795617c64688943 P 1f9cb72be98941b5a8e504cbe34132b9 [CANDIDATE]: Term 2 election: Requested vote from peers 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131:34471), 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129:46875), 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134:40141)
I20240501 07:59:10.306341 6576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9dc384dbac47dc9795617c64688943" candidate_uuid: "1f9cb72be98941b5a8e504cbe34132b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9"
I20240501 07:59:10.306465 6576 raft_consensus.cc:3032] T ff9dc384dbac47dc9795617c64688943 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 07:59:10.306529 6576 raft_consensus.cc:729] T ff9dc384dbac47dc9795617c64688943 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3b164cbbc00f4e8eb3254ddd349a3ff9, State: Running, Role: LEADER
I20240501 07:59:10.306490 8924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9dc384dbac47dc9795617c64688943" candidate_uuid: "1f9cb72be98941b5a8e504cbe34132b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "13d05ebdca8c4f489c597ce550dd773d"
I20240501 07:59:10.306625 8924 raft_consensus.cc:3037] T ff9dc384dbac47dc9795617c64688943 P 13d05ebdca8c4f489c597ce550dd773d [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:10.306715 6576 consensus_queue.cc:260] T ff9dc384dbac47dc9795617c64688943 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false } }
I20240501 07:59:10.306908 6576 raft_consensus.cc:3037] T ff9dc384dbac47dc9795617c64688943 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:10.307389 8924 raft_consensus.cc:2454] T ff9dc384dbac47dc9795617c64688943 P 13d05ebdca8c4f489c597ce550dd773d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f9cb72be98941b5a8e504cbe34132b9 in term 2.
I20240501 07:59:10.307777 6576 raft_consensus.cc:2454] T ff9dc384dbac47dc9795617c64688943 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f9cb72be98941b5a8e504cbe34132b9 in term 2.
I20240501 07:59:10.308090 6313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9dc384dbac47dc9795617c64688943" candidate_uuid: "1f9cb72be98941b5a8e504cbe34132b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f"
I20240501 07:59:10.308130 6644 leader_election.cc:302] T ff9dc384dbac47dc9795617c64688943 P 1f9cb72be98941b5a8e504cbe34132b9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 13d05ebdca8c4f489c597ce550dd773d, 1f9cb72be98941b5a8e504cbe34132b9, 3b164cbbc00f4e8eb3254ddd349a3ff9; no voters:
I20240501 07:59:10.308236 6313 raft_consensus.cc:3037] T ff9dc384dbac47dc9795617c64688943 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:10.308305 10324 raft_consensus.cc:2781] T ff9dc384dbac47dc9795617c64688943 P 1f9cb72be98941b5a8e504cbe34132b9 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 07:59:10.308398 10324 raft_consensus.cc:686] T ff9dc384dbac47dc9795617c64688943 P 1f9cb72be98941b5a8e504cbe34132b9 [term 2 LEADER]: Becoming Leader. State: Replica: 1f9cb72be98941b5a8e504cbe34132b9, State: Running, Role: LEADER
I20240501 07:59:10.308529 10324 consensus_queue.cc:237] T ff9dc384dbac47dc9795617c64688943 P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false } }
I20240501 07:59:10.309274 6313 raft_consensus.cc:2454] T ff9dc384dbac47dc9795617c64688943 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f9cb72be98941b5a8e504cbe34132b9 in term 2.
I20240501 07:59:10.311868 6168 catalog_manager.cc:5484] T ff9dc384dbac47dc9795617c64688943 P 1f9cb72be98941b5a8e504cbe34132b9 reported cstate change: term changed from 1 to 2, leader changed from 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131) to 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132). New cstate: current_term: 2 leader_uuid: "1f9cb72be98941b5a8e504cbe34132b9" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 07:59:10.331936 6168 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55950:
table { table_name: "rebalance_test_table_10" } modify_external_catalogs: true
I20240501 07:59:10.332094 6168 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55950:
table { table_name: "rebalance_test_table_10" } modify_external_catalogs: true
I20240501 07:59:10.334992 6168 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 0bc706e85ce74cef97a4827ea4d4aa65: Sending DeleteTablet for 4 replicas of tablet f9df6f3df2b94e5eb97dbff04dbe2ae1
I20240501 07:59:10.335355 6168 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 0bc706e85ce74cef97a4827ea4d4aa65: Sending DeleteTablet for 3 replicas of tablet 187940ba79f546b1ad79e7ec1de76442
I20240501 07:59:10.335602 6168 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 0bc706e85ce74cef97a4827ea4d4aa65: Sending DeleteTablet for 3 replicas of tablet 5122bc9a3cfb4b04aafc739256914bc2
I20240501 07:59:10.335602 6294 tablet_service.cc:1514] Processing DeleteTablet for tablet f9df6f3df2b94e5eb97dbff04dbe2ae1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:10 UTC) from {username='slave'} at 127.0.0.1:45480
I20240501 07:59:10.335842 6555 tablet_service.cc:1514] Processing DeleteTablet for tablet 187940ba79f546b1ad79e7ec1de76442 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:10 UTC) from {username='slave'} at 127.0.0.1:53874
I20240501 07:59:10.335861 9167 tablet_service.cc:1514] Processing DeleteTablet for tablet f9df6f3df2b94e5eb97dbff04dbe2ae1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:10 UTC) from {username='slave'} at 127.0.0.1:33594
I20240501 07:59:10.335901 6687 tablet_service.cc:1514] Processing DeleteTablet for tablet f9df6f3df2b94e5eb97dbff04dbe2ae1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:10 UTC) from {username='slave'} at 127.0.0.1:38524
I20240501 07:59:10.336014 9840 tablet_replica.cc:330] T 187940ba79f546b1ad79e7ec1de76442 P 3b164cbbc00f4e8eb3254ddd349a3ff9: stopping tablet replica
I20240501 07:59:10.336045 10175 tablet_replica.cc:330] T f9df6f3df2b94e5eb97dbff04dbe2ae1 P 83f20e84af4c40d09a4804b6b26e697f: stopping tablet replica
I20240501 07:59:10.336076 10309 tablet_replica.cc:330] T f9df6f3df2b94e5eb97dbff04dbe2ae1 P 8b8b3ad3aa8f4053b2167dd550f2086f: stopping tablet replica
I20240501 07:59:10.336153 9840 raft_consensus.cc:2229] T 187940ba79f546b1ad79e7ec1de76442 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:10.336185 10309 raft_consensus.cc:2229] T f9df6f3df2b94e5eb97dbff04dbe2ae1 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:10.336192 10175 raft_consensus.cc:2229] T f9df6f3df2b94e5eb97dbff04dbe2ae1 P 83f20e84af4c40d09a4804b6b26e697f [term 1 LEARNER]: Raft consensus shutting down.
I20240501 07:59:10.336287 10175 raft_consensus.cc:2258] T f9df6f3df2b94e5eb97dbff04dbe2ae1 P 83f20e84af4c40d09a4804b6b26e697f [term 1 LEARNER]: Raft consensus is shut down!
I20240501 07:59:10.336318 10309 raft_consensus.cc:2258] T f9df6f3df2b94e5eb97dbff04dbe2ae1 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:10.336319 9840 raft_consensus.cc:2258] T 187940ba79f546b1ad79e7ec1de76442 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:10.336485 10310 tablet_replica.cc:330] T f9df6f3df2b94e5eb97dbff04dbe2ae1 P 1f9cb72be98941b5a8e504cbe34132b9: stopping tablet replica
I20240501 07:59:10.336441 6425 tablet_service.cc:1514] Processing DeleteTablet for tablet f9df6f3df2b94e5eb97dbff04dbe2ae1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:10 UTC) from {username='slave'} at 127.0.0.1:45034
I20240501 07:59:10.336653 10075 tablet_replica.cc:330] T f9df6f3df2b94e5eb97dbff04dbe2ae1 P 8315d22c56f7439a9e1fc82f153f018c: stopping tablet replica
I20240501 07:59:10.336655 10310 raft_consensus.cc:2229] T f9df6f3df2b94e5eb97dbff04dbe2ae1 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:10.336781 10075 raft_consensus.cc:2229] T f9df6f3df2b94e5eb97dbff04dbe2ae1 P 8315d22c56f7439a9e1fc82f153f018c [term 1 LEADER]: Raft consensus shutting down.
I20240501 07:59:10.336804 10310 raft_consensus.cc:2258] T f9df6f3df2b94e5eb97dbff04dbe2ae1 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:10.337024 6294 tablet_service.cc:1514] Processing DeleteTablet for tablet 5122bc9a3cfb4b04aafc739256914bc2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:10 UTC) from {username='slave'} at 127.0.0.1:45480
I20240501 07:59:10.337062 10075 raft_consensus.cc:2258] T f9df6f3df2b94e5eb97dbff04dbe2ae1 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:10.337518 6687 tablet_service.cc:1514] Processing DeleteTablet for tablet 187940ba79f546b1ad79e7ec1de76442 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:10 UTC) from {username='slave'} at 127.0.0.1:38524
I20240501 07:59:10.337673 9840 ts_tablet_manager.cc:1875] T 187940ba79f546b1ad79e7ec1de76442 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:59:10.337800 10075 ts_tablet_manager.cc:1875] T f9df6f3df2b94e5eb97dbff04dbe2ae1 P 8315d22c56f7439a9e1fc82f153f018c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:59:10.337985 10175 ts_tablet_manager.cc:1875] T f9df6f3df2b94e5eb97dbff04dbe2ae1 P 83f20e84af4c40d09a4804b6b26e697f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:59:10.339469 6686 tablet_service.cc:1514] Processing DeleteTablet for tablet 5122bc9a3cfb4b04aafc739256914bc2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:10 UTC) from {username='slave'} at 127.0.0.1:38524
I20240501 07:59:10.339555 9840 ts_tablet_manager.cc:1888] T 187940ba79f546b1ad79e7ec1de76442 P 3b164cbbc00f4e8eb3254ddd349a3ff9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.6
I20240501 07:59:10.339622 9840 log.cc:1196] T 187940ba79f546b1ad79e7ec1de76442 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/187940ba79f546b1ad79e7ec1de76442
I20240501 07:59:10.339665 8905 tablet_service.cc:1514] Processing DeleteTablet for tablet 187940ba79f546b1ad79e7ec1de76442 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:10 UTC) from {username='slave'} at 127.0.0.1:47202
I20240501 07:59:10.339892 9840 ts_tablet_manager.cc:1909] T 187940ba79f546b1ad79e7ec1de76442 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Deleting consensus metadata
I20240501 07:59:10.339998 10075 ts_tablet_manager.cc:1888] T f9df6f3df2b94e5eb97dbff04dbe2ae1 P 8315d22c56f7439a9e1fc82f153f018c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:59:10.339998 10175 ts_tablet_manager.cc:1888] T f9df6f3df2b94e5eb97dbff04dbe2ae1 P 83f20e84af4c40d09a4804b6b26e697f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:59:10.340063 10175 log.cc:1196] T f9df6f3df2b94e5eb97dbff04dbe2ae1 P 83f20e84af4c40d09a4804b6b26e697f: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/f9df6f3df2b94e5eb97dbff04dbe2ae1
I20240501 07:59:10.340063 10075 log.cc:1196] T f9df6f3df2b94e5eb97dbff04dbe2ae1 P 8315d22c56f7439a9e1fc82f153f018c: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/f9df6f3df2b94e5eb97dbff04dbe2ae1
I20240501 07:59:10.340332 10175 ts_tablet_manager.cc:1909] T f9df6f3df2b94e5eb97dbff04dbe2ae1 P 83f20e84af4c40d09a4804b6b26e697f: Deleting consensus metadata
I20240501 07:59:10.340332 10075 ts_tablet_manager.cc:1909] T f9df6f3df2b94e5eb97dbff04dbe2ae1 P 8315d22c56f7439a9e1fc82f153f018c: Deleting consensus metadata
I20240501 07:59:10.341243 6160 catalog_manager.cc:4830] TS 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131:34471): tablet 187940ba79f546b1ad79e7ec1de76442 (table rebalance_test_table_10 [id=7b24d2d13db34edb849642d7d3eef300]) successfully deleted
I20240501 07:59:10.341487 10466 tablet_replica.cc:330] T 187940ba79f546b1ad79e7ec1de76442 P 13d05ebdca8c4f489c597ce550dd773d: stopping tablet replica
I20240501 07:59:10.341488 6159 catalog_manager.cc:4830] TS 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130:32941): tablet f9df6f3df2b94e5eb97dbff04dbe2ae1 (table rebalance_test_table_10 [id=7b24d2d13db34edb849642d7d3eef300]) successfully deleted
I20240501 07:59:10.341630 10466 raft_consensus.cc:2229] T 187940ba79f546b1ad79e7ec1de76442 P 13d05ebdca8c4f489c597ce550dd773d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:10.341782 10466 raft_consensus.cc:2258] T 187940ba79f546b1ad79e7ec1de76442 P 13d05ebdca8c4f489c597ce550dd773d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:10.341886 6425 tablet_service.cc:1514] Processing DeleteTablet for tablet 5122bc9a3cfb4b04aafc739256914bc2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:10 UTC) from {username='slave'} at 127.0.0.1:45034
I20240501 07:59:10.342031 10075 tablet_replica.cc:330] T 5122bc9a3cfb4b04aafc739256914bc2 P 8315d22c56f7439a9e1fc82f153f018c: stopping tablet replica
I20240501 07:59:10.342149 10075 raft_consensus.cc:2229] T 5122bc9a3cfb4b04aafc739256914bc2 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:10.342274 10075 raft_consensus.cc:2258] T 5122bc9a3cfb4b04aafc739256914bc2 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:10.342499 10466 ts_tablet_manager.cc:1875] T 187940ba79f546b1ad79e7ec1de76442 P 13d05ebdca8c4f489c597ce550dd773d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:59:10.342712 10075 ts_tablet_manager.cc:1875] T 5122bc9a3cfb4b04aafc739256914bc2 P 8315d22c56f7439a9e1fc82f153f018c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:59:10.344669 10075 ts_tablet_manager.cc:1888] T 5122bc9a3cfb4b04aafc739256914bc2 P 8315d22c56f7439a9e1fc82f153f018c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 07:59:10.344669 10466 ts_tablet_manager.cc:1888] T 187940ba79f546b1ad79e7ec1de76442 P 13d05ebdca8c4f489c597ce550dd773d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.6
I20240501 07:59:10.344735 10466 log.cc:1196] T 187940ba79f546b1ad79e7ec1de76442 P 13d05ebdca8c4f489c597ce550dd773d: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/187940ba79f546b1ad79e7ec1de76442
I20240501 07:59:10.344735 10075 log.cc:1196] T 5122bc9a3cfb4b04aafc739256914bc2 P 8315d22c56f7439a9e1fc82f153f018c: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/5122bc9a3cfb4b04aafc739256914bc2
I20240501 07:59:10.345019 10466 ts_tablet_manager.cc:1909] T 187940ba79f546b1ad79e7ec1de76442 P 13d05ebdca8c4f489c597ce550dd773d: Deleting consensus metadata
I20240501 07:59:10.345011 10075 ts_tablet_manager.cc:1909] T 5122bc9a3cfb4b04aafc739256914bc2 P 8315d22c56f7439a9e1fc82f153f018c: Deleting consensus metadata
I20240501 07:59:10.345844 10309 ts_tablet_manager.cc:1875] T f9df6f3df2b94e5eb97dbff04dbe2ae1 P 8b8b3ad3aa8f4053b2167dd550f2086f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:59:10.346127 6159 catalog_manager.cc:4830] TS 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130:32941): tablet 5122bc9a3cfb4b04aafc739256914bc2 (table rebalance_test_table_10 [id=7b24d2d13db34edb849642d7d3eef300]) successfully deleted
I20240501 07:59:10.346592 6159 catalog_manager.cc:4830] TS 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134:40141): tablet 187940ba79f546b1ad79e7ec1de76442 (table rebalance_test_table_10 [id=7b24d2d13db34edb849642d7d3eef300]) successfully deleted
I20240501 07:59:10.347735 10309 ts_tablet_manager.cc:1888] T f9df6f3df2b94e5eb97dbff04dbe2ae1 P 8b8b3ad3aa8f4053b2167dd550f2086f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:59:10.347787 10309 log.cc:1196] T f9df6f3df2b94e5eb97dbff04dbe2ae1 P 8b8b3ad3aa8f4053b2167dd550f2086f: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/f9df6f3df2b94e5eb97dbff04dbe2ae1
I20240501 07:59:10.348013 10309 ts_tablet_manager.cc:1909] T f9df6f3df2b94e5eb97dbff04dbe2ae1 P 8b8b3ad3aa8f4053b2167dd550f2086f: Deleting consensus metadata
I20240501 07:59:10.348652 10309 tablet_replica.cc:330] T 5122bc9a3cfb4b04aafc739256914bc2 P 8b8b3ad3aa8f4053b2167dd550f2086f: stopping tablet replica
I20240501 07:59:10.348790 10309 raft_consensus.cc:2229] T 5122bc9a3cfb4b04aafc739256914bc2 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 LEADER]: Raft consensus shutting down.
I20240501 07:59:10.348907 10310 ts_tablet_manager.cc:1875] T f9df6f3df2b94e5eb97dbff04dbe2ae1 P 1f9cb72be98941b5a8e504cbe34132b9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:59:10.348981 6157 catalog_manager.cc:4830] TS 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129:46875): tablet f9df6f3df2b94e5eb97dbff04dbe2ae1 (table rebalance_test_table_10 [id=7b24d2d13db34edb849642d7d3eef300]) successfully deleted
I20240501 07:59:10.348999 10309 raft_consensus.cc:2258] T 5122bc9a3cfb4b04aafc739256914bc2 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:10.349524 10309 ts_tablet_manager.cc:1875] T 5122bc9a3cfb4b04aafc739256914bc2 P 8b8b3ad3aa8f4053b2167dd550f2086f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:59:10.350358 6158 catalog_manager.cc:4830] TS 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): tablet f9df6f3df2b94e5eb97dbff04dbe2ae1 (table rebalance_test_table_10 [id=7b24d2d13db34edb849642d7d3eef300]) successfully deleted
I20240501 07:59:10.351020 10310 ts_tablet_manager.cc:1888] T f9df6f3df2b94e5eb97dbff04dbe2ae1 P 1f9cb72be98941b5a8e504cbe34132b9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:59:10.351092 10310 log.cc:1196] T f9df6f3df2b94e5eb97dbff04dbe2ae1 P 1f9cb72be98941b5a8e504cbe34132b9: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/f9df6f3df2b94e5eb97dbff04dbe2ae1
I20240501 07:59:10.351197 10309 ts_tablet_manager.cc:1888] T 5122bc9a3cfb4b04aafc739256914bc2 P 8b8b3ad3aa8f4053b2167dd550f2086f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 07:59:10.351266 10309 log.cc:1196] T 5122bc9a3cfb4b04aafc739256914bc2 P 8b8b3ad3aa8f4053b2167dd550f2086f: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/5122bc9a3cfb4b04aafc739256914bc2
I20240501 07:59:10.351352 10310 ts_tablet_manager.cc:1909] T f9df6f3df2b94e5eb97dbff04dbe2ae1 P 1f9cb72be98941b5a8e504cbe34132b9: Deleting consensus metadata
I20240501 07:59:10.351505 10309 ts_tablet_manager.cc:1909] T 5122bc9a3cfb4b04aafc739256914bc2 P 8b8b3ad3aa8f4053b2167dd550f2086f: Deleting consensus metadata
I20240501 07:59:10.351945 10310 tablet_replica.cc:330] T 187940ba79f546b1ad79e7ec1de76442 P 1f9cb72be98941b5a8e504cbe34132b9: stopping tablet replica
I20240501 07:59:10.352011 6159 catalog_manager.cc:4830] TS 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132:36773): tablet f9df6f3df2b94e5eb97dbff04dbe2ae1 (table rebalance_test_table_10 [id=7b24d2d13db34edb849642d7d3eef300]) successfully deleted
I20240501 07:59:10.352154 10310 raft_consensus.cc:2229] T 187940ba79f546b1ad79e7ec1de76442 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 LEADER]: Raft consensus shutting down.
I20240501 07:59:10.352206 6157 catalog_manager.cc:4830] TS 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129:46875): tablet 5122bc9a3cfb4b04aafc739256914bc2 (table rebalance_test_table_10 [id=7b24d2d13db34edb849642d7d3eef300]) successfully deleted
I20240501 07:59:10.352351 10310 raft_consensus.cc:2258] T 187940ba79f546b1ad79e7ec1de76442 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:10.352952 10310 ts_tablet_manager.cc:1875] T 187940ba79f546b1ad79e7ec1de76442 P 1f9cb72be98941b5a8e504cbe34132b9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:59:10.354769 10310 ts_tablet_manager.cc:1888] T 187940ba79f546b1ad79e7ec1de76442 P 1f9cb72be98941b5a8e504cbe34132b9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.6
I20240501 07:59:10.354832 10310 log.cc:1196] T 187940ba79f546b1ad79e7ec1de76442 P 1f9cb72be98941b5a8e504cbe34132b9: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/187940ba79f546b1ad79e7ec1de76442
I20240501 07:59:10.355041 10310 ts_tablet_manager.cc:1909] T 187940ba79f546b1ad79e7ec1de76442 P 1f9cb72be98941b5a8e504cbe34132b9: Deleting consensus metadata
I20240501 07:59:10.355636 6168 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55950:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 07:59:10.356000 10310 tablet_replica.cc:330] T 5122bc9a3cfb4b04aafc739256914bc2 P 1f9cb72be98941b5a8e504cbe34132b9: stopping tablet replica
I20240501 07:59:10.356125 10310 raft_consensus.cc:2229] T 5122bc9a3cfb4b04aafc739256914bc2 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:10.356114 6159 catalog_manager.cc:4830] TS 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132:36773): tablet 187940ba79f546b1ad79e7ec1de76442 (table rebalance_test_table_10 [id=7b24d2d13db34edb849642d7d3eef300]) successfully deleted
I20240501 07:59:10.356249 10310 raft_consensus.cc:2258] T 5122bc9a3cfb4b04aafc739256914bc2 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:10.356721 10310 ts_tablet_manager.cc:1875] T 5122bc9a3cfb4b04aafc739256914bc2 P 1f9cb72be98941b5a8e504cbe34132b9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:59:10.358678 10310 ts_tablet_manager.cc:1888] T 5122bc9a3cfb4b04aafc739256914bc2 P 1f9cb72be98941b5a8e504cbe34132b9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 07:59:10.358810 10310 log.cc:1196] T 5122bc9a3cfb4b04aafc739256914bc2 P 1f9cb72be98941b5a8e504cbe34132b9: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/5122bc9a3cfb4b04aafc739256914bc2
I20240501 07:59:10.359099 10310 ts_tablet_manager.cc:1909] T 5122bc9a3cfb4b04aafc739256914bc2 P 1f9cb72be98941b5a8e504cbe34132b9: Deleting consensus metadata
I20240501 07:59:10.361160 6159 catalog_manager.cc:4830] TS 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132:36773): tablet 5122bc9a3cfb4b04aafc739256914bc2 (table rebalance_test_table_10 [id=7b24d2d13db34edb849642d7d3eef300]) successfully deleted
I20240501 07:59:10.361840 10373 tablet.cc:1722] T 51b9e79f78894a7fa240271f9a36f72b P 83f20e84af4c40d09a4804b6b26e697f: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 07:59:10.362322 10378 tablet.cc:1722] T 51b9e79f78894a7fa240271f9a36f72b P 13d05ebdca8c4f489c597ce550dd773d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 07:59:10.363483 10375 tablet.cc:1722] T 51b9e79f78894a7fa240271f9a36f72b P a505e7cb133c4392bfda8a4a20874ae9: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 07:59:10.366950 6168 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 0bc706e85ce74cef97a4827ea4d4aa65: rebalancer_dynamic_table_new_name [id=4fa2f76148be476390669c4d79dbb6bd] alter complete (version 3)
I20240501 07:59:10.369416 9400 consensus_queue.cc:1035] T 239ee596e62d47169eeb6670dc62d59c P 13d05ebdca8c4f489c597ce550dd773d [LEADER]: Connected to new peer: Peer: permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:10.371416 6707 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c3a9036ccfee45498bca9681518d3699"
dest_uuid: "1f9cb72be98941b5a8e504cbe34132b9"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:38414
I20240501 07:59:10.371599 6707 raft_consensus.cc:595] T c3a9036ccfee45498bca9681518d3699 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 LEADER]: Received request to transfer leadership
I20240501 07:59:10.373672 10478 consensus_queue.cc:1035] T 239ee596e62d47169eeb6670dc62d59c P 13d05ebdca8c4f489c597ce550dd773d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:10.406477 6445 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "bab34f083c8045e4a6df373db5ebd54b"
dest_uuid: "8315d22c56f7439a9e1fc82f153f018c"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:36486
I20240501 07:59:10.406652 6445 raft_consensus.cc:595] T bab34f083c8045e4a6df373db5ebd54b P 8315d22c56f7439a9e1fc82f153f018c [term 1 LEADER]: Received request to transfer leadership
I20240501 07:59:10.414503 6311 consensus_queue.cc:237] T 2398430a23094711a13a86cc88e27a78 P 8b8b3ad3aa8f4053b2167dd550f2086f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: true } }
I20240501 07:59:10.416126 6576 raft_consensus.cc:1261] T 2398430a23094711a13a86cc88e27a78 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Refusing update from remote peer 8b8b3ad3aa8f4053b2167dd550f2086f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:59:10.416210 6445 raft_consensus.cc:1261] T 2398430a23094711a13a86cc88e27a78 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Refusing update from remote peer 8b8b3ad3aa8f4053b2167dd550f2086f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 07:59:10.416298 6248 consensus_peers.cc:485] T 2398430a23094711a13a86cc88e27a78 P 8b8b3ad3aa8f4053b2167dd550f2086f -> Peer 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133:39859): Couldn't send request to peer 74077b61d6f54342bf9736b4a9f2eade. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2398430a23094711a13a86cc88e27a78. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:10.416584 9373 consensus_queue.cc:1035] T 2398430a23094711a13a86cc88e27a78 P 8b8b3ad3aa8f4053b2167dd550f2086f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:10.416921 9373 consensus_queue.cc:1035] T 2398430a23094711a13a86cc88e27a78 P 8b8b3ad3aa8f4053b2167dd550f2086f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:10.418362 9373 raft_consensus.cc:2932] T 2398430a23094711a13a86cc88e27a78 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: true } } }
I20240501 07:59:10.418787 6576 raft_consensus.cc:2932] T 2398430a23094711a13a86cc88e27a78 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: true } } }
I20240501 07:59:10.420810 6445 raft_consensus.cc:2932] T 2398430a23094711a13a86cc88e27a78 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: true } } }
I20240501 07:59:10.422927 6164 catalog_manager.cc:5484] T 2398430a23094711a13a86cc88e27a78 P 8b8b3ad3aa8f4053b2167dd550f2086f reported cstate change: config changed from index -1 to 4, NON_VOTER 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 07:59:10.435410 6576 consensus_queue.cc:237] T 33b3834db0cd4407ae7c26e044459e87 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: true } }
I20240501 07:59:10.436841 6445 raft_consensus.cc:1261] T 33b3834db0cd4407ae7c26e044459e87 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Refusing update from remote peer 3b164cbbc00f4e8eb3254ddd349a3ff9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 07:59:10.437109 6511 consensus_peers.cc:485] T 33b3834db0cd4407ae7c26e044459e87 P 3b164cbbc00f4e8eb3254ddd349a3ff9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 33b3834db0cd4407ae7c26e044459e87. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:10.437234 6311 raft_consensus.cc:1261] T 33b3834db0cd4407ae7c26e044459e87 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Refusing update from remote peer 3b164cbbc00f4e8eb3254ddd349a3ff9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:59:10.437650 8680 consensus_queue.cc:1035] T 33b3834db0cd4407ae7c26e044459e87 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:10.437850 9507 consensus_queue.cc:1035] T 33b3834db0cd4407ae7c26e044459e87 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:10.439204 9990 raft_consensus.cc:979] T c3a9036ccfee45498bca9681518d3699 P 1f9cb72be98941b5a8e504cbe34132b9: : Instructing follower 8315d22c56f7439a9e1fc82f153f018c to start an election
I20240501 07:59:10.439314 9990 raft_consensus.cc:1067] T c3a9036ccfee45498bca9681518d3699 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 LEADER]: Signalling peer 8315d22c56f7439a9e1fc82f153f018c to start an election
I20240501 07:59:10.439653 10046 raft_consensus.cc:2932] T 33b3834db0cd4407ae7c26e044459e87 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: true } } }
I20240501 07:59:10.439846 6445 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c3a9036ccfee45498bca9681518d3699"
dest_uuid: "8315d22c56f7439a9e1fc82f153f018c"
from {username='slave'} at 127.0.99.132:60527
I20240501 07:59:10.439982 6445 raft_consensus.cc:482] T c3a9036ccfee45498bca9681518d3699 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:59:10.440037 6445 raft_consensus.cc:3037] T c3a9036ccfee45498bca9681518d3699 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:10.440093 6444 raft_consensus.cc:2932] T 33b3834db0cd4407ae7c26e044459e87 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: true } } }
I20240501 07:59:10.440366 6311 raft_consensus.cc:2932] T 33b3834db0cd4407ae7c26e044459e87 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: true } } }
I20240501 07:59:10.441093 6445 raft_consensus.cc:504] T c3a9036ccfee45498bca9681518d3699 P 8315d22c56f7439a9e1fc82f153f018c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { replace: true } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } }
I20240501 07:59:10.441608 6445 leader_election.cc:288] T c3a9036ccfee45498bca9681518d3699 P 8315d22c56f7439a9e1fc82f153f018c [CANDIDATE]: Term 2 election: Requested vote from peers 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131:34471), 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132:36773), 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133:39859)
I20240501 07:59:10.442415 8793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3a9036ccfee45498bca9681518d3699" candidate_uuid: "8315d22c56f7439a9e1fc82f153f018c" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "74077b61d6f54342bf9736b4a9f2eade"
I20240501 07:59:10.442481 6576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3a9036ccfee45498bca9681518d3699" candidate_uuid: "8315d22c56f7439a9e1fc82f153f018c" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9"
I20240501 07:59:10.442540 8793 raft_consensus.cc:3037] T c3a9036ccfee45498bca9681518d3699 P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:10.442597 6576 raft_consensus.cc:3037] T c3a9036ccfee45498bca9681518d3699 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:10.443466 6576 raft_consensus.cc:2454] T c3a9036ccfee45498bca9681518d3699 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8315d22c56f7439a9e1fc82f153f018c in term 2.
I20240501 07:59:10.443537 8793 raft_consensus.cc:2454] T c3a9036ccfee45498bca9681518d3699 P 74077b61d6f54342bf9736b4a9f2eade [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8315d22c56f7439a9e1fc82f153f018c in term 2.
I20240501 07:59:10.443408 6164 catalog_manager.cc:5484] T 33b3834db0cd4407ae7c26e044459e87 P 8b8b3ad3aa8f4053b2167dd550f2086f reported cstate change: config changed from index -1 to 4, NON_VOTER a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: true } } }
I20240501 07:59:10.444371 6379 leader_election.cc:302] T c3a9036ccfee45498bca9681518d3699 P 8315d22c56f7439a9e1fc82f153f018c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 3b164cbbc00f4e8eb3254ddd349a3ff9, 74077b61d6f54342bf9736b4a9f2eade, 8315d22c56f7439a9e1fc82f153f018c; no voters:
I20240501 07:59:10.445227 10111 raft_consensus.cc:2781] T c3a9036ccfee45498bca9681518d3699 P 8315d22c56f7439a9e1fc82f153f018c [term 2 FOLLOWER]: Leader election won for term 2
I20240501 07:59:10.445318 10111 raft_consensus.cc:686] T c3a9036ccfee45498bca9681518d3699 P 8315d22c56f7439a9e1fc82f153f018c [term 2 LEADER]: Becoming Leader. State: Replica: 8315d22c56f7439a9e1fc82f153f018c, State: Running, Role: LEADER
I20240501 07:59:10.445456 10111 consensus_queue.cc:237] T c3a9036ccfee45498bca9681518d3699 P 8315d22c56f7439a9e1fc82f153f018c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { replace: true } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } }
I20240501 07:59:10.445714 6707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3a9036ccfee45498bca9681518d3699" candidate_uuid: "8315d22c56f7439a9e1fc82f153f018c" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "1f9cb72be98941b5a8e504cbe34132b9"
I20240501 07:59:10.445833 6707 raft_consensus.cc:3032] T c3a9036ccfee45498bca9681518d3699 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 07:59:10.445870 6707 raft_consensus.cc:729] T c3a9036ccfee45498bca9681518d3699 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1f9cb72be98941b5a8e504cbe34132b9, State: Running, Role: LEADER
I20240501 07:59:10.446023 6707 consensus_queue.cc:260] T c3a9036ccfee45498bca9681518d3699 P 1f9cb72be98941b5a8e504cbe34132b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { replace: true } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } }
I20240501 07:59:10.446218 6707 raft_consensus.cc:3037] T c3a9036ccfee45498bca9681518d3699 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:10.447124 6707 raft_consensus.cc:2454] T c3a9036ccfee45498bca9681518d3699 P 1f9cb72be98941b5a8e504cbe34132b9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8315d22c56f7439a9e1fc82f153f018c in term 2.
I20240501 07:59:10.449141 6164 catalog_manager.cc:5484] T c3a9036ccfee45498bca9681518d3699 P 8315d22c56f7439a9e1fc82f153f018c reported cstate change: term changed from 1 to 2, leader changed from 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132) to 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130). New cstate: current_term: 2 leader_uuid: "8315d22c56f7439a9e1fc82f153f018c" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 07:59:10.457696 8793 consensus_queue.cc:237] T 6292ff0c7e624dd7b24715b83a30fff8 P 74077b61d6f54342bf9736b4a9f2eade [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { replace: true } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { promote: true } }
I20240501 07:59:10.459513 6576 raft_consensus.cc:1261] T 6292ff0c7e624dd7b24715b83a30fff8 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Refusing update from remote peer 74077b61d6f54342bf9736b4a9f2eade: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:10.459628 9188 raft_consensus.cc:1261] T 6292ff0c7e624dd7b24715b83a30fff8 P 83f20e84af4c40d09a4804b6b26e697f [term 1 FOLLOWER]: Refusing update from remote peer 74077b61d6f54342bf9736b4a9f2eade: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:10.459895 10528 consensus_queue.cc:1035] T 6292ff0c7e624dd7b24715b83a30fff8 P 74077b61d6f54342bf9736b4a9f2eade [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 07:59:10.460109 10069 consensus_queue.cc:1035] T 6292ff0c7e624dd7b24715b83a30fff8 P 74077b61d6f54342bf9736b4a9f2eade [LEADER]: Connected to new peer: Peer: permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 07:59:10.460314 8729 consensus_peers.cc:485] T 6292ff0c7e624dd7b24715b83a30fff8 P 74077b61d6f54342bf9736b4a9f2eade -> Peer 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132:36773): Couldn't send request to peer 1f9cb72be98941b5a8e504cbe34132b9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6292ff0c7e624dd7b24715b83a30fff8. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:10.462114 10528 raft_consensus.cc:2932] T 6292ff0c7e624dd7b24715b83a30fff8 P 74077b61d6f54342bf9736b4a9f2eade [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { replace: true } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { promote: true } } }
I20240501 07:59:10.462587 6576 raft_consensus.cc:2932] T 6292ff0c7e624dd7b24715b83a30fff8 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { replace: true } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { promote: true } } }
I20240501 07:59:10.463450 9188 raft_consensus.cc:2932] T 6292ff0c7e624dd7b24715b83a30fff8 P 83f20e84af4c40d09a4804b6b26e697f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { replace: true } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { promote: true } } }
I20240501 07:59:10.465873 6164 catalog_manager.cc:5484] T 6292ff0c7e624dd7b24715b83a30fff8 P 74077b61d6f54342bf9736b4a9f2eade reported cstate change: config changed from index -1 to 2, NON_VOTER 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132) added. New cstate: current_term: 1 leader_uuid: "74077b61d6f54342bf9736b4a9f2eade" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 07:59:10.468760 6168 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55950:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 07:59:10.468873 6168 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55950:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 07:59:10.471189 6168 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 0bc706e85ce74cef97a4827ea4d4aa65: Sending DeleteTablet for 3 replicas of tablet 51b9e79f78894a7fa240271f9a36f72b
I20240501 07:59:10.471853 8905 tablet_service.cc:1514] Processing DeleteTablet for tablet 51b9e79f78894a7fa240271f9a36f72b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:10 UTC) from {username='slave'} at 127.0.0.1:47202
I20240501 07:59:10.472101 10466 tablet_replica.cc:330] T 51b9e79f78894a7fa240271f9a36f72b P 13d05ebdca8c4f489c597ce550dd773d: stopping tablet replica
I20240501 07:59:10.472158 9036 tablet_service.cc:1514] Processing DeleteTablet for tablet 51b9e79f78894a7fa240271f9a36f72b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:10 UTC) from {username='slave'} at 127.0.0.1:44236
I20240501 07:59:10.472266 10466 raft_consensus.cc:2229] T 51b9e79f78894a7fa240271f9a36f72b P 13d05ebdca8c4f489c597ce550dd773d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:10.472409 10466 raft_consensus.cc:2258] T 51b9e79f78894a7fa240271f9a36f72b P 13d05ebdca8c4f489c597ce550dd773d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:10.472529 10532 tablet_replica.cc:330] T 51b9e79f78894a7fa240271f9a36f72b P a505e7cb133c4392bfda8a4a20874ae9: stopping tablet replica
I20240501 07:59:10.472584 9167 tablet_service.cc:1514] Processing DeleteTablet for tablet 51b9e79f78894a7fa240271f9a36f72b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:10 UTC) from {username='slave'} at 127.0.0.1:33594
I20240501 07:59:10.472652 10532 raft_consensus.cc:2229] T 51b9e79f78894a7fa240271f9a36f72b P a505e7cb133c4392bfda8a4a20874ae9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:10.472766 10532 raft_consensus.cc:2258] T 51b9e79f78894a7fa240271f9a36f72b P a505e7cb133c4392bfda8a4a20874ae9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:10.472913 10175 tablet_replica.cc:330] T 51b9e79f78894a7fa240271f9a36f72b P 83f20e84af4c40d09a4804b6b26e697f: stopping tablet replica
I20240501 07:59:10.473057 10175 raft_consensus.cc:2229] T 51b9e79f78894a7fa240271f9a36f72b P 83f20e84af4c40d09a4804b6b26e697f [term 1 LEADER]: Raft consensus shutting down.
I20240501 07:59:10.473258 10175 raft_consensus.cc:2258] T 51b9e79f78894a7fa240271f9a36f72b P 83f20e84af4c40d09a4804b6b26e697f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:10.473889 10532 ts_tablet_manager.cc:1875] T 51b9e79f78894a7fa240271f9a36f72b P a505e7cb133c4392bfda8a4a20874ae9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:59:10.474169 10466 ts_tablet_manager.cc:1875] T 51b9e79f78894a7fa240271f9a36f72b P 13d05ebdca8c4f489c597ce550dd773d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:59:10.476295 10466 ts_tablet_manager.cc:1888] T 51b9e79f78894a7fa240271f9a36f72b P 13d05ebdca8c4f489c597ce550dd773d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:59:10.476296 10532 ts_tablet_manager.cc:1888] T 51b9e79f78894a7fa240271f9a36f72b P a505e7cb133c4392bfda8a4a20874ae9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:59:10.476377 10466 log.cc:1196] T 51b9e79f78894a7fa240271f9a36f72b P 13d05ebdca8c4f489c597ce550dd773d: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/51b9e79f78894a7fa240271f9a36f72b
I20240501 07:59:10.476377 10532 log.cc:1196] T 51b9e79f78894a7fa240271f9a36f72b P a505e7cb133c4392bfda8a4a20874ae9: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/51b9e79f78894a7fa240271f9a36f72b
I20240501 07:59:10.476662 10466 ts_tablet_manager.cc:1909] T 51b9e79f78894a7fa240271f9a36f72b P 13d05ebdca8c4f489c597ce550dd773d: Deleting consensus metadata
I20240501 07:59:10.476675 10532 ts_tablet_manager.cc:1909] T 51b9e79f78894a7fa240271f9a36f72b P a505e7cb133c4392bfda8a4a20874ae9: Deleting consensus metadata
I20240501 07:59:10.477478 10175 ts_tablet_manager.cc:1875] T 51b9e79f78894a7fa240271f9a36f72b P 83f20e84af4c40d09a4804b6b26e697f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:59:10.478210 6158 catalog_manager.cc:4830] TS a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): tablet 51b9e79f78894a7fa240271f9a36f72b (table rebalancer_dynamic_table_new_name [id=4fa2f76148be476390669c4d79dbb6bd]) successfully deleted
I20240501 07:59:10.479430 10175 ts_tablet_manager.cc:1888] T 51b9e79f78894a7fa240271f9a36f72b P 83f20e84af4c40d09a4804b6b26e697f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:59:10.479512 10175 log.cc:1196] T 51b9e79f78894a7fa240271f9a36f72b P 83f20e84af4c40d09a4804b6b26e697f: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/51b9e79f78894a7fa240271f9a36f72b
I20240501 07:59:10.479794 10175 ts_tablet_manager.cc:1909] T 51b9e79f78894a7fa240271f9a36f72b P 83f20e84af4c40d09a4804b6b26e697f: Deleting consensus metadata
I20240501 07:59:10.480522 6158 catalog_manager.cc:4830] TS 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): tablet 51b9e79f78894a7fa240271f9a36f72b (table rebalancer_dynamic_table_new_name [id=4fa2f76148be476390669c4d79dbb6bd]) successfully deleted
I20240501 07:59:10.481112 6159 catalog_manager.cc:4830] TS 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134:40141): tablet 51b9e79f78894a7fa240271f9a36f72b (table rebalancer_dynamic_table_new_name [id=4fa2f76148be476390669c4d79dbb6bd]) successfully deleted
I20240501 07:59:10.482589 6576 consensus_queue.cc:237] T 0b680bccf110443e9495942863364ff9 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: true } }
W20240501 07:59:10.484200 6510 consensus_peers.cc:485] T 0b680bccf110443e9495942863364ff9 P 3b164cbbc00f4e8eb3254ddd349a3ff9 -> Peer 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133:39859): Couldn't send request to peer 74077b61d6f54342bf9736b4a9f2eade. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0b680bccf110443e9495942863364ff9. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:10.484236 6708 raft_consensus.cc:1261] T 0b680bccf110443e9495942863364ff9 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Refusing update from remote peer 3b164cbbc00f4e8eb3254ddd349a3ff9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:59:10.484273 6311 raft_consensus.cc:1261] T 0b680bccf110443e9495942863364ff9 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Refusing update from remote peer 3b164cbbc00f4e8eb3254ddd349a3ff9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:59:10.484637 9507 consensus_queue.cc:1035] T 0b680bccf110443e9495942863364ff9 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:10.484778 10237 consensus_queue.cc:1035] T 0b680bccf110443e9495942863364ff9 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:10.486552 9507 raft_consensus.cc:2932] T 0b680bccf110443e9495942863364ff9 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: true } } }
I20240501 07:59:10.487144 6311 raft_consensus.cc:2932] T 0b680bccf110443e9495942863364ff9 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: true } } }
I20240501 07:59:10.487144 6708 raft_consensus.cc:2932] T 0b680bccf110443e9495942863364ff9 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: true } } }
I20240501 07:59:10.489766 6168 catalog_manager.cc:5484] T 0b680bccf110443e9495942863364ff9 P 3b164cbbc00f4e8eb3254ddd349a3ff9 reported cstate change: config changed from index -1 to 4, NON_VOTER 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 07:59:10.502804 6708 consensus_queue.cc:237] T b284ed2cb0264ced99a4ffd90983d902 P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { replace: true } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: true } }
I20240501 07:59:10.504483 9055 raft_consensus.cc:1203] T 3745acee2b5543b697a3c8f45d2618bb P a505e7cb133c4392bfda8a4a20874ae9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 07:59:10.505357 6311 raft_consensus.cc:1261] T b284ed2cb0264ced99a4ffd90983d902 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Refusing update from remote peer 1f9cb72be98941b5a8e504cbe34132b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:59:10.505792 10324 consensus_queue.cc:1035] T b284ed2cb0264ced99a4ffd90983d902 P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 07:59:10.506430 6643 consensus_peers.cc:485] T b284ed2cb0264ced99a4ffd90983d902 P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134:40141): Couldn't send request to peer 13d05ebdca8c4f489c597ce550dd773d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b284ed2cb0264ced99a4ffd90983d902. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:10.506572 6576 raft_consensus.cc:1261] T b284ed2cb0264ced99a4ffd90983d902 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Refusing update from remote peer 1f9cb72be98941b5a8e504cbe34132b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:59:10.506938 10324 consensus_queue.cc:1035] T b284ed2cb0264ced99a4ffd90983d902 P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:10.507967 10288 raft_consensus.cc:2932] T b284ed2cb0264ced99a4ffd90983d902 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { replace: true } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: true } } }
I20240501 07:59:10.508399 6311 raft_consensus.cc:2932] T b284ed2cb0264ced99a4ffd90983d902 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { replace: true } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: true } } }
I20240501 07:59:10.509088 6576 raft_consensus.cc:2932] T b284ed2cb0264ced99a4ffd90983d902 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { replace: true } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: true } } }
I20240501 07:59:10.512208 6166 catalog_manager.cc:5484] T b284ed2cb0264ced99a4ffd90983d902 P 3b164cbbc00f4e8eb3254ddd349a3ff9 reported cstate change: config changed from index -1 to 4, NON_VOTER 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "1f9cb72be98941b5a8e504cbe34132b9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { replace: true } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: true } } }
I20240501 07:59:10.520777 6311 consensus_queue.cc:237] T f68dbc1f65eb4f1b91a62e2de82a7676 P 8b8b3ad3aa8f4053b2167dd550f2086f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: true } }
I20240501 07:59:10.522135 6445 raft_consensus.cc:1261] T f68dbc1f65eb4f1b91a62e2de82a7676 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Refusing update from remote peer 8b8b3ad3aa8f4053b2167dd550f2086f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 07:59:10.522294 6249 consensus_peers.cc:485] T f68dbc1f65eb4f1b91a62e2de82a7676 P 8b8b3ad3aa8f4053b2167dd550f2086f -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f68dbc1f65eb4f1b91a62e2de82a7676. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:10.522483 6576 raft_consensus.cc:1261] T f68dbc1f65eb4f1b91a62e2de82a7676 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Refusing update from remote peer 8b8b3ad3aa8f4053b2167dd550f2086f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:59:10.522773 9964 consensus_queue.cc:1035] T f68dbc1f65eb4f1b91a62e2de82a7676 P 8b8b3ad3aa8f4053b2167dd550f2086f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:10.523005 9373 consensus_queue.cc:1035] T f68dbc1f65eb4f1b91a62e2de82a7676 P 8b8b3ad3aa8f4053b2167dd550f2086f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:10.524644 9964 raft_consensus.cc:2932] T f68dbc1f65eb4f1b91a62e2de82a7676 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: true } } }
I20240501 07:59:10.525056 6445 raft_consensus.cc:2932] T f68dbc1f65eb4f1b91a62e2de82a7676 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: true } } }
I20240501 07:59:10.525385 6576 raft_consensus.cc:2932] T f68dbc1f65eb4f1b91a62e2de82a7676 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: true } } }
I20240501 07:59:10.528153 6171 catalog_manager.cc:5484] T f68dbc1f65eb4f1b91a62e2de82a7676 P 8315d22c56f7439a9e1fc82f153f018c reported cstate change: config changed from index -1 to 4, NON_VOTER 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: true } } }
I20240501 07:59:10.541456 6311 consensus_queue.cc:237] T 6c3c0d6fe41f4ae196ae29beb9b68125 P 8b8b3ad3aa8f4053b2167dd550f2086f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { replace: true } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: true } }
I20240501 07:59:10.542892 6708 raft_consensus.cc:1261] T 6c3c0d6fe41f4ae196ae29beb9b68125 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Refusing update from remote peer 8b8b3ad3aa8f4053b2167dd550f2086f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:59:10.542897 6576 raft_consensus.cc:1261] T 6c3c0d6fe41f4ae196ae29beb9b68125 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Refusing update from remote peer 8b8b3ad3aa8f4053b2167dd550f2086f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 07:59:10.543020 6249 consensus_peers.cc:485] T 6c3c0d6fe41f4ae196ae29beb9b68125 P 8b8b3ad3aa8f4053b2167dd550f2086f -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6c3c0d6fe41f4ae196ae29beb9b68125. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:10.543351 10040 consensus_queue.cc:1035] T 6c3c0d6fe41f4ae196ae29beb9b68125 P 8b8b3ad3aa8f4053b2167dd550f2086f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:10.543550 9964 consensus_queue.cc:1035] T 6c3c0d6fe41f4ae196ae29beb9b68125 P 8b8b3ad3aa8f4053b2167dd550f2086f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:10.545464 10040 raft_consensus.cc:2932] T 6c3c0d6fe41f4ae196ae29beb9b68125 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { replace: true } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: true } } }
I20240501 07:59:10.546139 6577 raft_consensus.cc:2932] T 6c3c0d6fe41f4ae196ae29beb9b68125 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { replace: true } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: true } } }
I20240501 07:59:10.547200 6708 raft_consensus.cc:2932] T 6c3c0d6fe41f4ae196ae29beb9b68125 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { replace: true } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: true } } }
I20240501 07:59:10.549718 6166 catalog_manager.cc:5484] T 6c3c0d6fe41f4ae196ae29beb9b68125 P 8b8b3ad3aa8f4053b2167dd550f2086f reported cstate change: config changed from index -1 to 4, NON_VOTER a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 07:59:10.557143 6445 consensus_queue.cc:237] T e92d6aeeb3df4ea3852c6ebbed0e07c5 P 8315d22c56f7439a9e1fc82f153f018c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: true } }
I20240501 07:59:10.558518 6577 raft_consensus.cc:1261] T e92d6aeeb3df4ea3852c6ebbed0e07c5 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Refusing update from remote peer 8315d22c56f7439a9e1fc82f153f018c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:59:10.558511 6311 raft_consensus.cc:1261] T e92d6aeeb3df4ea3852c6ebbed0e07c5 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Refusing update from remote peer 8315d22c56f7439a9e1fc82f153f018c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 07:59:10.558813 6381 consensus_peers.cc:485] T e92d6aeeb3df4ea3852c6ebbed0e07c5 P 8315d22c56f7439a9e1fc82f153f018c -> Peer 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134:40141): Couldn't send request to peer 13d05ebdca8c4f489c597ce550dd773d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e92d6aeeb3df4ea3852c6ebbed0e07c5. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:10.558909 10111 consensus_queue.cc:1035] T e92d6aeeb3df4ea3852c6ebbed0e07c5 P 8315d22c56f7439a9e1fc82f153f018c [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:10.559090 8512 consensus_queue.cc:1035] T e92d6aeeb3df4ea3852c6ebbed0e07c5 P 8315d22c56f7439a9e1fc82f153f018c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:10.560866 9116 raft_consensus.cc:2932] T e92d6aeeb3df4ea3852c6ebbed0e07c5 P 8315d22c56f7439a9e1fc82f153f018c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: true } } }
I20240501 07:59:10.561327 6577 raft_consensus.cc:2932] T e92d6aeeb3df4ea3852c6ebbed0e07c5 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: true } } }
I20240501 07:59:10.561409 6311 raft_consensus.cc:2932] T e92d6aeeb3df4ea3852c6ebbed0e07c5 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: true } } }
I20240501 07:59:10.564074 6168 catalog_manager.cc:5484] T e92d6aeeb3df4ea3852c6ebbed0e07c5 P 8315d22c56f7439a9e1fc82f153f018c reported cstate change: config changed from index -1 to 4, NON_VOTER 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "8315d22c56f7439a9e1fc82f153f018c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 07:59:10.572602 8924 consensus_queue.cc:237] T fb6b46072a8d46e18b53b9f8de702859 P 13d05ebdca8c4f489c597ce550dd773d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { replace: true } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { promote: true } }
I20240501 07:59:10.573554 6166 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55950:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 07:59:10.574405 8793 raft_consensus.cc:1261] T fb6b46072a8d46e18b53b9f8de702859 P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Refusing update from remote peer 13d05ebdca8c4f489c597ce550dd773d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:10.574538 9188 raft_consensus.cc:1261] T fb6b46072a8d46e18b53b9f8de702859 P 83f20e84af4c40d09a4804b6b26e697f [term 1 FOLLOWER]: Refusing update from remote peer 13d05ebdca8c4f489c597ce550dd773d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:10.574805 10235 consensus_queue.cc:1035] T fb6b46072a8d46e18b53b9f8de702859 P 13d05ebdca8c4f489c597ce550dd773d [LEADER]: Connected to new peer: Peer: permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 07:59:10.574971 9400 consensus_queue.cc:1035] T fb6b46072a8d46e18b53b9f8de702859 P 13d05ebdca8c4f489c597ce550dd773d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 07:59:10.575276 8858 consensus_peers.cc:485] T fb6b46072a8d46e18b53b9f8de702859 P 13d05ebdca8c4f489c597ce550dd773d -> Peer 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129:46875): Couldn't send request to peer 8b8b3ad3aa8f4053b2167dd550f2086f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fb6b46072a8d46e18b53b9f8de702859. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:10.577306 10235 raft_consensus.cc:2932] T fb6b46072a8d46e18b53b9f8de702859 P 13d05ebdca8c4f489c597ce550dd773d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { replace: true } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { promote: true } } }
I20240501 07:59:10.579449 9188 raft_consensus.cc:2932] T fb6b46072a8d46e18b53b9f8de702859 P 83f20e84af4c40d09a4804b6b26e697f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { replace: true } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { promote: true } } }
I20240501 07:59:10.579547 8793 raft_consensus.cc:2932] T fb6b46072a8d46e18b53b9f8de702859 P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { replace: true } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { promote: true } } }
I20240501 07:59:10.580750 6166 catalog_manager.cc:5484] T fb6b46072a8d46e18b53b9f8de702859 P 13d05ebdca8c4f489c597ce550dd773d reported cstate change: config changed from index -1 to 2, NON_VOTER 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129) added. New cstate: current_term: 1 leader_uuid: "13d05ebdca8c4f489c597ce550dd773d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 07:59:10.584503 6555 tablet_service.cc:1467] Processing CreateTablet for tablet dbbae25711d94ed29223004d019ef377 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=2c07df4ff4ae47538ff574e5adb1fd0b]), partition=
I20240501 07:59:10.584786 6555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbbae25711d94ed29223004d019ef377. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:10.585005 9167 tablet_service.cc:1467] Processing CreateTablet for tablet dbbae25711d94ed29223004d019ef377 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=2c07df4ff4ae47538ff574e5adb1fd0b]), partition=
I20240501 07:59:10.585268 9167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbbae25711d94ed29223004d019ef377. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:10.587026 10588 tablet_bootstrap.cc:492] T dbbae25711d94ed29223004d019ef377 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Bootstrap starting.
I20240501 07:59:10.587155 9036 tablet_service.cc:1467] Processing CreateTablet for tablet dbbae25711d94ed29223004d019ef377 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=2c07df4ff4ae47538ff574e5adb1fd0b]), partition=
I20240501 07:59:10.587399 9036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbbae25711d94ed29223004d019ef377. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:10.587503 10589 tablet_bootstrap.cc:492] T dbbae25711d94ed29223004d019ef377 P 83f20e84af4c40d09a4804b6b26e697f: Bootstrap starting.
I20240501 07:59:10.588052 10588 tablet_bootstrap.cc:654] T dbbae25711d94ed29223004d019ef377 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:10.588292 10589 tablet_bootstrap.cc:654] T dbbae25711d94ed29223004d019ef377 P 83f20e84af4c40d09a4804b6b26e697f: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:10.589614 10592 tablet_bootstrap.cc:492] T dbbae25711d94ed29223004d019ef377 P a505e7cb133c4392bfda8a4a20874ae9: Bootstrap starting.
I20240501 07:59:10.589797 10588 tablet_bootstrap.cc:492] T dbbae25711d94ed29223004d019ef377 P 3b164cbbc00f4e8eb3254ddd349a3ff9: No bootstrap required, opened a new log
I20240501 07:59:10.589835 10589 tablet_bootstrap.cc:492] T dbbae25711d94ed29223004d019ef377 P 83f20e84af4c40d09a4804b6b26e697f: No bootstrap required, opened a new log
I20240501 07:59:10.589869 10588 ts_tablet_manager.cc:1367] T dbbae25711d94ed29223004d019ef377 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 07:59:10.589911 10589 ts_tablet_manager.cc:1367] T dbbae25711d94ed29223004d019ef377 P 83f20e84af4c40d09a4804b6b26e697f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:59:10.590298 10588 raft_consensus.cc:348] T dbbae25711d94ed29223004d019ef377 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } }
I20240501 07:59:10.590329 10589 raft_consensus.cc:348] T dbbae25711d94ed29223004d019ef377 P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } }
I20240501 07:59:10.590427 10589 raft_consensus.cc:374] T dbbae25711d94ed29223004d019ef377 P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:10.590492 10589 raft_consensus.cc:729] T dbbae25711d94ed29223004d019ef377 P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83f20e84af4c40d09a4804b6b26e697f, State: Initialized, Role: FOLLOWER
I20240501 07:59:10.590559 10588 raft_consensus.cc:374] T dbbae25711d94ed29223004d019ef377 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:10.590608 10588 raft_consensus.cc:729] T dbbae25711d94ed29223004d019ef377 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b164cbbc00f4e8eb3254ddd349a3ff9, State: Initialized, Role: FOLLOWER
I20240501 07:59:10.590632 10589 consensus_queue.cc:260] T dbbae25711d94ed29223004d019ef377 P 83f20e84af4c40d09a4804b6b26e697f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } }
I20240501 07:59:10.590716 10588 consensus_queue.cc:260] T dbbae25711d94ed29223004d019ef377 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } }
I20240501 07:59:10.590912 10589 ts_tablet_manager.cc:1398] T dbbae25711d94ed29223004d019ef377 P 83f20e84af4c40d09a4804b6b26e697f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:10.591007 10588 ts_tablet_manager.cc:1398] T dbbae25711d94ed29223004d019ef377 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:10.592252 10592 tablet_bootstrap.cc:654] T dbbae25711d94ed29223004d019ef377 P a505e7cb133c4392bfda8a4a20874ae9: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:10.593721 10592 tablet_bootstrap.cc:492] T dbbae25711d94ed29223004d019ef377 P a505e7cb133c4392bfda8a4a20874ae9: No bootstrap required, opened a new log
I20240501 07:59:10.593828 10592 ts_tablet_manager.cc:1367] T dbbae25711d94ed29223004d019ef377 P a505e7cb133c4392bfda8a4a20874ae9: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240501 07:59:10.594767 10592 raft_consensus.cc:348] T dbbae25711d94ed29223004d019ef377 P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } }
I20240501 07:59:10.594889 10592 raft_consensus.cc:374] T dbbae25711d94ed29223004d019ef377 P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:10.594923 10592 raft_consensus.cc:729] T dbbae25711d94ed29223004d019ef377 P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a505e7cb133c4392bfda8a4a20874ae9, State: Initialized, Role: FOLLOWER
I20240501 07:59:10.595049 10592 consensus_queue.cc:260] T dbbae25711d94ed29223004d019ef377 P a505e7cb133c4392bfda8a4a20874ae9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } }
I20240501 07:59:10.595358 10592 ts_tablet_manager.cc:1398] T dbbae25711d94ed29223004d019ef377 P a505e7cb133c4392bfda8a4a20874ae9: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20240501 07:59:10.598402 6313 consensus_queue.cc:237] T 2e540f389b2a4aef93a020f549d11b67 P 8b8b3ad3aa8f4053b2167dd550f2086f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: true } }
I20240501 07:59:10.599843 6577 raft_consensus.cc:1261] T 2e540f389b2a4aef93a020f549d11b67 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Refusing update from remote peer 8b8b3ad3aa8f4053b2167dd550f2086f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:59:10.599844 6445 raft_consensus.cc:1261] T 2e540f389b2a4aef93a020f549d11b67 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Refusing update from remote peer 8b8b3ad3aa8f4053b2167dd550f2086f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:59:10.600196 10040 consensus_queue.cc:1035] T 2e540f389b2a4aef93a020f549d11b67 P 8b8b3ad3aa8f4053b2167dd550f2086f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 07:59:10.600369 6249 consensus_peers.cc:485] T 2e540f389b2a4aef93a020f549d11b67 P 8b8b3ad3aa8f4053b2167dd550f2086f -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2e540f389b2a4aef93a020f549d11b67. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:10.600375 9373 consensus_queue.cc:1035] T 2e540f389b2a4aef93a020f549d11b67 P 8b8b3ad3aa8f4053b2167dd550f2086f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:10.602402 9373 raft_consensus.cc:2932] T 2e540f389b2a4aef93a020f549d11b67 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: true } } }
I20240501 07:59:10.603078 6444 raft_consensus.cc:2932] T 2e540f389b2a4aef93a020f549d11b67 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: true } } }
I20240501 07:59:10.603910 6577 raft_consensus.cc:2932] T 2e540f389b2a4aef93a020f549d11b67 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: true } } }
I20240501 07:59:10.606200 6164 catalog_manager.cc:5484] T 2e540f389b2a4aef93a020f549d11b67 P 8b8b3ad3aa8f4053b2167dd550f2086f reported cstate change: config changed from index -1 to 4, NON_VOTER 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 07:59:10.615080 6577 consensus_queue.cc:237] T 114b6dbafa1c4c44ac42f1d2293e3125 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: true } }
I20240501 07:59:10.616583 6708 raft_consensus.cc:1261] T 114b6dbafa1c4c44ac42f1d2293e3125 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Refusing update from remote peer 3b164cbbc00f4e8eb3254ddd349a3ff9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:59:10.616583 6444 raft_consensus.cc:1261] T 114b6dbafa1c4c44ac42f1d2293e3125 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Refusing update from remote peer 3b164cbbc00f4e8eb3254ddd349a3ff9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 07:59:10.616905 6511 consensus_peers.cc:485] T 114b6dbafa1c4c44ac42f1d2293e3125 P 3b164cbbc00f4e8eb3254ddd349a3ff9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 114b6dbafa1c4c44ac42f1d2293e3125. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:10.617393 8680 consensus_queue.cc:1035] T 114b6dbafa1c4c44ac42f1d2293e3125 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:10.617604 10237 consensus_queue.cc:1035] T 114b6dbafa1c4c44ac42f1d2293e3125 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:10.619400 9507 raft_consensus.cc:2932] T 114b6dbafa1c4c44ac42f1d2293e3125 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: true } } }
I20240501 07:59:10.619817 6444 raft_consensus.cc:2932] T 114b6dbafa1c4c44ac42f1d2293e3125 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: true } } }
I20240501 07:59:10.619894 6708 raft_consensus.cc:2932] T 114b6dbafa1c4c44ac42f1d2293e3125 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: true } } }
I20240501 07:59:10.622588 6166 catalog_manager.cc:5484] T 114b6dbafa1c4c44ac42f1d2293e3125 P 3b164cbbc00f4e8eb3254ddd349a3ff9 reported cstate change: config changed from index -1 to 4, NON_VOTER a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 07:59:10.647095 10196 consensus_queue.cc:1035] T 2976d6353e8947b7b6c37f72efbeeac4 P 83f20e84af4c40d09a4804b6b26e697f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:10.651546 9673 consensus_queue.cc:1035] T 2976d6353e8947b7b6c37f72efbeeac4 P 83f20e84af4c40d09a4804b6b26e697f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:10.661569 8924 raft_consensus.cc:1203] T 72feac2a993141168ac9399d9361ef17 P 13d05ebdca8c4f489c597ce550dd773d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 07:59:10.686144 10352 ts_tablet_manager.cc:902] T adf207c019994dd49671bab4145fc268 P a505e7cb133c4392bfda8a4a20874ae9: Initiating tablet copy from peer 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130:32941)
I20240501 07:59:10.686728 10352 tablet_copy_client.cc:308] T adf207c019994dd49671bab4145fc268 P a505e7cb133c4392bfda8a4a20874ae9: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:32941
I20240501 07:59:10.688109 6466 tablet_copy_service.cc:140] P 8315d22c56f7439a9e1fc82f153f018c: Received BeginTabletCopySession request for tablet adf207c019994dd49671bab4145fc268 from peer a505e7cb133c4392bfda8a4a20874ae9 ({username='slave'} at 127.0.99.135:36631)
I20240501 07:59:10.688233 6466 tablet_copy_service.cc:161] P 8315d22c56f7439a9e1fc82f153f018c: Beginning new tablet copy session on tablet adf207c019994dd49671bab4145fc268 from peer a505e7cb133c4392bfda8a4a20874ae9 at {username='slave'} at 127.0.99.135:36631: session id = a505e7cb133c4392bfda8a4a20874ae9-adf207c019994dd49671bab4145fc268
I20240501 07:59:10.689239 6466 tablet_copy_source_session.cc:205] T adf207c019994dd49671bab4145fc268 P 8315d22c56f7439a9e1fc82f153f018c: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:10.690135 10352 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet adf207c019994dd49671bab4145fc268. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:10.692499 10352 tablet_copy_client.cc:783] T adf207c019994dd49671bab4145fc268 P a505e7cb133c4392bfda8a4a20874ae9: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:10.692674 10352 tablet_copy_client.cc:647] T adf207c019994dd49671bab4145fc268 P a505e7cb133c4392bfda8a4a20874ae9: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:10.694015 10352 tablet_copy_client.cc:523] T adf207c019994dd49671bab4145fc268 P a505e7cb133c4392bfda8a4a20874ae9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:10.695384 10352 tablet_bootstrap.cc:492] T adf207c019994dd49671bab4145fc268 P a505e7cb133c4392bfda8a4a20874ae9: Bootstrap starting.
I20240501 07:59:10.704321 10528 consensus_queue.cc:1035] T 2f9e82c3f1df4d1292e37e9cea07186e P 74077b61d6f54342bf9736b4a9f2eade [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:10.710724 10403 consensus_queue.cc:1035] T 2f9e82c3f1df4d1292e37e9cea07186e P 74077b61d6f54342bf9736b4a9f2eade [LEADER]: Connected to new peer: Peer: permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:10.721079 10352 tablet_bootstrap.cc:492] T adf207c019994dd49671bab4145fc268 P a505e7cb133c4392bfda8a4a20874ae9: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:10.721412 10352 tablet_bootstrap.cc:492] T adf207c019994dd49671bab4145fc268 P a505e7cb133c4392bfda8a4a20874ae9: Bootstrap complete.
I20240501 07:59:10.721617 10352 ts_tablet_manager.cc:1367] T adf207c019994dd49671bab4145fc268 P a505e7cb133c4392bfda8a4a20874ae9: Time spent bootstrapping tablet: real 0.026s user 0.025s sys 0.000s
I20240501 07:59:10.722091 10352 raft_consensus.cc:348] T adf207c019994dd49671bab4145fc268 P a505e7cb133c4392bfda8a4a20874ae9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { replace: true } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: true } }
I20240501 07:59:10.722229 10352 raft_consensus.cc:729] T adf207c019994dd49671bab4145fc268 P a505e7cb133c4392bfda8a4a20874ae9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a505e7cb133c4392bfda8a4a20874ae9, State: Initialized, Role: LEARNER
I20240501 07:59:10.722389 10352 consensus_queue.cc:260] T adf207c019994dd49671bab4145fc268 P a505e7cb133c4392bfda8a4a20874ae9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { replace: true } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: true } }
I20240501 07:59:10.722697 10352 ts_tablet_manager.cc:1398] T adf207c019994dd49671bab4145fc268 P a505e7cb133c4392bfda8a4a20874ae9: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:59:10.723356 6466 tablet_copy_service.cc:342] P 8315d22c56f7439a9e1fc82f153f018c: Request end of tablet copy session a505e7cb133c4392bfda8a4a20874ae9-adf207c019994dd49671bab4145fc268 received from {username='slave'} at 127.0.99.135:36631
I20240501 07:59:10.723444 6466 tablet_copy_service.cc:433] P 8315d22c56f7439a9e1fc82f153f018c: ending tablet copy session a505e7cb133c4392bfda8a4a20874ae9-adf207c019994dd49671bab4145fc268 on tablet adf207c019994dd49671bab4145fc268 with peer a505e7cb133c4392bfda8a4a20874ae9
I20240501 07:59:10.736122 8512 raft_consensus.cc:979] T bab34f083c8045e4a6df373db5ebd54b P 8315d22c56f7439a9e1fc82f153f018c: : Instructing follower 8b8b3ad3aa8f4053b2167dd550f2086f to start an election
I20240501 07:59:10.736274 8512 raft_consensus.cc:1067] T bab34f083c8045e4a6df373db5ebd54b P 8315d22c56f7439a9e1fc82f153f018c [term 1 LEADER]: Signalling peer 8b8b3ad3aa8f4053b2167dd550f2086f to start an election
I20240501 07:59:10.736891 6313 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "bab34f083c8045e4a6df373db5ebd54b"
dest_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f"
from {username='slave'} at 127.0.99.130:37291
I20240501 07:59:10.737073 6313 raft_consensus.cc:482] T bab34f083c8045e4a6df373db5ebd54b P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:59:10.737147 6313 raft_consensus.cc:3037] T bab34f083c8045e4a6df373db5ebd54b P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:10.738178 6313 raft_consensus.cc:504] T bab34f083c8045e4a6df373db5ebd54b P 8b8b3ad3aa8f4053b2167dd550f2086f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false } }
I20240501 07:59:10.738659 6313 leader_election.cc:288] T bab34f083c8045e4a6df373db5ebd54b P 8b8b3ad3aa8f4053b2167dd550f2086f [CANDIDATE]: Term 2 election: Requested vote from peers 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131:34471), 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130:32941), 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134:40141)
I20240501 07:59:10.739228 6577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bab34f083c8045e4a6df373db5ebd54b" candidate_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9"
I20240501 07:59:10.739365 6577 raft_consensus.cc:3037] T bab34f083c8045e4a6df373db5ebd54b P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:10.739354 8924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bab34f083c8045e4a6df373db5ebd54b" candidate_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "13d05ebdca8c4f489c597ce550dd773d"
I20240501 07:59:10.739485 8924 raft_consensus.cc:3037] T bab34f083c8045e4a6df373db5ebd54b P 13d05ebdca8c4f489c597ce550dd773d [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:10.740509 6577 raft_consensus.cc:2454] T bab34f083c8045e4a6df373db5ebd54b P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b8b3ad3aa8f4053b2167dd550f2086f in term 2.
I20240501 07:59:10.740511 8924 raft_consensus.cc:2454] T bab34f083c8045e4a6df373db5ebd54b P 13d05ebdca8c4f489c597ce550dd773d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b8b3ad3aa8f4053b2167dd550f2086f in term 2.
I20240501 07:59:10.741039 6251 leader_election.cc:302] T bab34f083c8045e4a6df373db5ebd54b P 8b8b3ad3aa8f4053b2167dd550f2086f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 13d05ebdca8c4f489c597ce550dd773d, 3b164cbbc00f4e8eb3254ddd349a3ff9, 8b8b3ad3aa8f4053b2167dd550f2086f; no voters:
I20240501 07:59:10.741236 9373 raft_consensus.cc:2781] T bab34f083c8045e4a6df373db5ebd54b P 8b8b3ad3aa8f4053b2167dd550f2086f [term 2 FOLLOWER]: Leader election won for term 2
I20240501 07:59:10.741329 9373 raft_consensus.cc:686] T bab34f083c8045e4a6df373db5ebd54b P 8b8b3ad3aa8f4053b2167dd550f2086f [term 2 LEADER]: Becoming Leader. State: Replica: 8b8b3ad3aa8f4053b2167dd550f2086f, State: Running, Role: LEADER
I20240501 07:59:10.741483 9373 consensus_queue.cc:237] T bab34f083c8045e4a6df373db5ebd54b P 8b8b3ad3aa8f4053b2167dd550f2086f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false } }
I20240501 07:59:10.742177 6444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bab34f083c8045e4a6df373db5ebd54b" candidate_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "8315d22c56f7439a9e1fc82f153f018c"
I20240501 07:59:10.742305 6444 raft_consensus.cc:3032] T bab34f083c8045e4a6df373db5ebd54b P 8315d22c56f7439a9e1fc82f153f018c [term 1 LEADER]: Stepping down as leader of term 1
I20240501 07:59:10.742343 6444 raft_consensus.cc:729] T bab34f083c8045e4a6df373db5ebd54b P 8315d22c56f7439a9e1fc82f153f018c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8315d22c56f7439a9e1fc82f153f018c, State: Running, Role: LEADER
I20240501 07:59:10.742931 6444 consensus_queue.cc:260] T bab34f083c8045e4a6df373db5ebd54b P 8315d22c56f7439a9e1fc82f153f018c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false } }
I20240501 07:59:10.743194 6444 raft_consensus.cc:3037] T bab34f083c8045e4a6df373db5ebd54b P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:10.743964 6164 catalog_manager.cc:5484] T bab34f083c8045e4a6df373db5ebd54b P 8b8b3ad3aa8f4053b2167dd550f2086f reported cstate change: term changed from 1 to 2, leader changed from 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130) to 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129). New cstate: current_term: 2 leader_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 07:59:10.744238 6444 raft_consensus.cc:2454] T bab34f083c8045e4a6df373db5ebd54b P 8315d22c56f7439a9e1fc82f153f018c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b8b3ad3aa8f4053b2167dd550f2086f in term 2.
I20240501 07:59:10.811069 10717 ts_tablet_manager.cc:902] T 2398430a23094711a13a86cc88e27a78 P 74077b61d6f54342bf9736b4a9f2eade: Initiating tablet copy from peer 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129:46875)
I20240501 07:59:10.811580 10717 tablet_copy_client.cc:308] T 2398430a23094711a13a86cc88e27a78 P 74077b61d6f54342bf9736b4a9f2eade: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:46875
I20240501 07:59:10.812134 6335 tablet_copy_service.cc:140] P 8b8b3ad3aa8f4053b2167dd550f2086f: Received BeginTabletCopySession request for tablet 2398430a23094711a13a86cc88e27a78 from peer 74077b61d6f54342bf9736b4a9f2eade ({username='slave'} at 127.0.99.133:50111)
I20240501 07:59:10.812263 6335 tablet_copy_service.cc:161] P 8b8b3ad3aa8f4053b2167dd550f2086f: Beginning new tablet copy session on tablet 2398430a23094711a13a86cc88e27a78 from peer 74077b61d6f54342bf9736b4a9f2eade at {username='slave'} at 127.0.99.133:50111: session id = 74077b61d6f54342bf9736b4a9f2eade-2398430a23094711a13a86cc88e27a78
I20240501 07:59:10.813373 6335 tablet_copy_source_session.cc:205] T 2398430a23094711a13a86cc88e27a78 P 8b8b3ad3aa8f4053b2167dd550f2086f: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:10.814410 10717 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2398430a23094711a13a86cc88e27a78. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:10.815861 6168 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55950:
name: "rebalancer_extra_table_8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 07:59:10.817525 10237 raft_consensus.cc:482] T dbbae25711d94ed29223004d019ef377 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:10.817637 10237 raft_consensus.cc:504] T dbbae25711d94ed29223004d019ef377 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } }
I20240501 07:59:10.818058 10237 leader_election.cc:288] T dbbae25711d94ed29223004d019ef377 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607), 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149)
I20240501 07:59:10.820698 9188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbbae25711d94ed29223004d019ef377" candidate_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83f20e84af4c40d09a4804b6b26e697f" is_pre_election: true
I20240501 07:59:10.820844 9188 raft_consensus.cc:2454] T dbbae25711d94ed29223004d019ef377 P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b164cbbc00f4e8eb3254ddd349a3ff9 in term 0.
I20240501 07:59:10.821259 9055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbbae25711d94ed29223004d019ef377" candidate_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a505e7cb133c4392bfda8a4a20874ae9" is_pre_election: true
I20240501 07:59:10.821388 9055 raft_consensus.cc:2454] T dbbae25711d94ed29223004d019ef377 P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b164cbbc00f4e8eb3254ddd349a3ff9 in term 0.
I20240501 07:59:10.822593 6511 leader_election.cc:302] T dbbae25711d94ed29223004d019ef377 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b164cbbc00f4e8eb3254ddd349a3ff9, a505e7cb133c4392bfda8a4a20874ae9; no voters:
I20240501 07:59:10.824518 10046 raft_consensus.cc:2781] T dbbae25711d94ed29223004d019ef377 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:10.824584 10046 raft_consensus.cc:482] T dbbae25711d94ed29223004d019ef377 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:10.824620 10046 raft_consensus.cc:3037] T dbbae25711d94ed29223004d019ef377 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:10.825793 10046 raft_consensus.cc:504] T dbbae25711d94ed29223004d019ef377 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } }
I20240501 07:59:10.826153 10046 leader_election.cc:288] T dbbae25711d94ed29223004d019ef377 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [CANDIDATE]: Term 1 election: Requested vote from peers a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607), 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149)
I20240501 07:59:10.826866 8905 tablet_service.cc:1467] Processing CreateTablet for tablet 72aa4dc7664b48b7a73d34feba7d44d3 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=7f1dc10bb484406aa39a54213d4854b9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:10.827167 8905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72aa4dc7664b48b7a73d34feba7d44d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:10.827334 6294 tablet_service.cc:1467] Processing CreateTablet for tablet ab1666f1e5204416adb435d9bbe80433 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=7f1dc10bb484406aa39a54213d4854b9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:10.827490 8757 tablet_service.cc:1467] Processing CreateTablet for tablet 72aa4dc7664b48b7a73d34feba7d44d3 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=7f1dc10bb484406aa39a54213d4854b9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:10.827589 6294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab1666f1e5204416adb435d9bbe80433. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:10.827739 8757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72aa4dc7664b48b7a73d34feba7d44d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:10.829655 6686 tablet_service.cc:1467] Processing CreateTablet for tablet 542c9ac08f7d4bec9b15a309c2e21364 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=7f1dc10bb484406aa39a54213d4854b9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:10.829948 6686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 542c9ac08f7d4bec9b15a309c2e21364. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:10.829999 8904 tablet_service.cc:1467] Processing CreateTablet for tablet 542c9ac08f7d4bec9b15a309c2e21364 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=7f1dc10bb484406aa39a54213d4854b9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:10.830226 8904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 542c9ac08f7d4bec9b15a309c2e21364. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:10.830420 8757 tablet_service.cc:1467] Processing CreateTablet for tablet ab1666f1e5204416adb435d9bbe80433 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=7f1dc10bb484406aa39a54213d4854b9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:10.830651 8757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab1666f1e5204416adb435d9bbe80433. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:10.833520 10717 tablet_copy_client.cc:783] T 2398430a23094711a13a86cc88e27a78 P 74077b61d6f54342bf9736b4a9f2eade: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:10.833676 10717 tablet_copy_client.cc:647] T 2398430a23094711a13a86cc88e27a78 P 74077b61d6f54342bf9736b4a9f2eade: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:10.833777 9167 tablet_service.cc:1467] Processing CreateTablet for tablet ab1666f1e5204416adb435d9bbe80433 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=7f1dc10bb484406aa39a54213d4854b9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:10.834008 10744 tablet_bootstrap.cc:492] T 542c9ac08f7d4bec9b15a309c2e21364 P 13d05ebdca8c4f489c597ce550dd773d: Bootstrap starting.
I20240501 07:59:10.834035 9167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab1666f1e5204416adb435d9bbe80433. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:10.834908 10744 tablet_bootstrap.cc:654] T 542c9ac08f7d4bec9b15a309c2e21364 P 13d05ebdca8c4f489c597ce550dd773d: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:10.834918 10717 tablet_copy_client.cc:523] T 2398430a23094711a13a86cc88e27a78 P 74077b61d6f54342bf9736b4a9f2eade: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:10.836056 10589 tablet_bootstrap.cc:492] T ab1666f1e5204416adb435d9bbe80433 P 83f20e84af4c40d09a4804b6b26e697f: Bootstrap starting.
I20240501 07:59:10.836097 10717 tablet_bootstrap.cc:492] T 2398430a23094711a13a86cc88e27a78 P 74077b61d6f54342bf9736b4a9f2eade: Bootstrap starting.
I20240501 07:59:10.836560 9036 tablet_service.cc:1467] Processing CreateTablet for tablet 72aa4dc7664b48b7a73d34feba7d44d3 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=7f1dc10bb484406aa39a54213d4854b9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:10.836828 9036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72aa4dc7664b48b7a73d34feba7d44d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:10.836918 10589 tablet_bootstrap.cc:654] T ab1666f1e5204416adb435d9bbe80433 P 83f20e84af4c40d09a4804b6b26e697f: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:10.838626 10592 tablet_bootstrap.cc:492] T 72aa4dc7664b48b7a73d34feba7d44d3 P a505e7cb133c4392bfda8a4a20874ae9: Bootstrap starting.
I20240501 07:59:10.838795 9035 tablet_service.cc:1467] Processing CreateTablet for tablet 542c9ac08f7d4bec9b15a309c2e21364 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=7f1dc10bb484406aa39a54213d4854b9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:10.839004 9035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 542c9ac08f7d4bec9b15a309c2e21364. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:10.839521 10592 tablet_bootstrap.cc:654] T 72aa4dc7664b48b7a73d34feba7d44d3 P a505e7cb133c4392bfda8a4a20874ae9: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:10.841197 10745 tablet_bootstrap.cc:492] T 542c9ac08f7d4bec9b15a309c2e21364 P 1f9cb72be98941b5a8e504cbe34132b9: Bootstrap starting.
I20240501 07:59:10.842119 10745 tablet_bootstrap.cc:654] T 542c9ac08f7d4bec9b15a309c2e21364 P 1f9cb72be98941b5a8e504cbe34132b9: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:10.843991 6168 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55950:
table { table_name: "rebalance_test_table_12" } modify_external_catalogs: true
I20240501 07:59:10.844141 6168 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55950:
table { table_name: "rebalance_test_table_12" } modify_external_catalogs: true
I20240501 07:59:10.846532 9055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbbae25711d94ed29223004d019ef377" candidate_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a505e7cb133c4392bfda8a4a20874ae9"
I20240501 07:59:10.846673 9055 raft_consensus.cc:3037] T dbbae25711d94ed29223004d019ef377 P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:10.847064 6168 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 0bc706e85ce74cef97a4827ea4d4aa65: Sending DeleteTablet for 4 replicas of tablet bab34f083c8045e4a6df373db5ebd54b
I20240501 07:59:10.847420 6168 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 0bc706e85ce74cef97a4827ea4d4aa65: Sending DeleteTablet for 4 replicas of tablet adf207c019994dd49671bab4145fc268
I20240501 07:59:10.847661 9055 raft_consensus.cc:2454] T dbbae25711d94ed29223004d019ef377 P a505e7cb133c4392bfda8a4a20874ae9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b164cbbc00f4e8eb3254ddd349a3ff9 in term 1.
I20240501 07:59:10.847756 6168 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 0bc706e85ce74cef97a4827ea4d4aa65: Sending DeleteTablet for 4 replicas of tablet 0b680bccf110443e9495942863364ff9
I20240501 07:59:10.847910 9188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbbae25711d94ed29223004d019ef377" candidate_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83f20e84af4c40d09a4804b6b26e697f"
I20240501 07:59:10.848033 9188 raft_consensus.cc:3037] T dbbae25711d94ed29223004d019ef377 P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:10.848755 6293 tablet_service.cc:1514] Processing DeleteTablet for tablet adf207c019994dd49671bab4145fc268 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:10 UTC) from {username='slave'} at 127.0.0.1:45480
I20240501 07:59:10.848815 6295 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b680bccf110443e9495942863364ff9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:10 UTC) from {username='slave'} at 127.0.0.1:45480
I20240501 07:59:10.848815 6294 tablet_service.cc:1514] Processing DeleteTablet for tablet bab34f083c8045e4a6df373db5ebd54b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:10 UTC) from {username='slave'} at 127.0.0.1:45480
I20240501 07:59:10.848973 10309 tablet_replica.cc:330] T adf207c019994dd49671bab4145fc268 P 8b8b3ad3aa8f4053b2167dd550f2086f: stopping tablet replica
I20240501 07:59:10.849071 9188 raft_consensus.cc:2454] T dbbae25711d94ed29223004d019ef377 P 83f20e84af4c40d09a4804b6b26e697f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b164cbbc00f4e8eb3254ddd349a3ff9 in term 1.
I20240501 07:59:10.849072 6424 tablet_service.cc:1514] Processing DeleteTablet for tablet adf207c019994dd49671bab4145fc268 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:10 UTC) from {username='slave'} at 127.0.0.1:45034
I20240501 07:59:10.849138 10309 raft_consensus.cc:2229] T adf207c019994dd49671bab4145fc268 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:10.849292 10309 raft_consensus.cc:2258] T adf207c019994dd49671bab4145fc268 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:10.849285 6425 tablet_service.cc:1514] Processing DeleteTablet for tablet bab34f083c8045e4a6df373db5ebd54b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:10 UTC) from {username='slave'} at 127.0.0.1:45034
I20240501 07:59:10.849527 10745 tablet_bootstrap.cc:492] T 542c9ac08f7d4bec9b15a309c2e21364 P 1f9cb72be98941b5a8e504cbe34132b9: No bootstrap required, opened a new log
I20240501 07:59:10.849588 10751 tablet_replica.cc:330] T adf207c019994dd49671bab4145fc268 P 8315d22c56f7439a9e1fc82f153f018c: stopping tablet replica
I20240501 07:59:10.849630 10745 ts_tablet_manager.cc:1367] T 542c9ac08f7d4bec9b15a309c2e21364 P 1f9cb72be98941b5a8e504cbe34132b9: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240501 07:59:10.849956 8757 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b680bccf110443e9495942863364ff9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:10 UTC) from {username='slave'} at 127.0.0.1:43136
I20240501 07:59:10.849936 10751 raft_consensus.cc:2229] T adf207c019994dd49671bab4145fc268 P 8315d22c56f7439a9e1fc82f153f018c [term 1 LEADER]: Raft consensus shutting down.
I20240501 07:59:10.850028 10309 ts_tablet_manager.cc:1875] T adf207c019994dd49671bab4145fc268 P 8b8b3ad3aa8f4053b2167dd550f2086f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:59:10.850206 10745 raft_consensus.cc:348] T 542c9ac08f7d4bec9b15a309c2e21364 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } }
I20240501 07:59:10.850417 10745 raft_consensus.cc:374] T 542c9ac08f7d4bec9b15a309c2e21364 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:10.850505 10745 raft_consensus.cc:729] T 542c9ac08f7d4bec9b15a309c2e21364 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f9cb72be98941b5a8e504cbe34132b9, State: Initialized, Role: FOLLOWER
I20240501 07:59:10.850556 10751 raft_consensus.cc:2258] T adf207c019994dd49671bab4145fc268 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:10.850708 10745 consensus_queue.cc:260] T 542c9ac08f7d4bec9b15a309c2e21364 P 1f9cb72be98941b5a8e504cbe34132b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } }
I20240501 07:59:10.851085 10745 ts_tablet_manager.cc:1398] T 542c9ac08f7d4bec9b15a309c2e21364 P 1f9cb72be98941b5a8e504cbe34132b9: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240501 07:59:10.851905 10751 ts_tablet_manager.cc:1875] T adf207c019994dd49671bab4145fc268 P 8315d22c56f7439a9e1fc82f153f018c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:59:10.852191 10309 ts_tablet_manager.cc:1888] T adf207c019994dd49671bab4145fc268 P 8b8b3ad3aa8f4053b2167dd550f2086f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:59:10.852264 10309 log.cc:1196] T adf207c019994dd49671bab4145fc268 P 8b8b3ad3aa8f4053b2167dd550f2086f: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/adf207c019994dd49671bab4145fc268
I20240501 07:59:10.852566 10309 ts_tablet_manager.cc:1909] T adf207c019994dd49671bab4145fc268 P 8b8b3ad3aa8f4053b2167dd550f2086f: Deleting consensus metadata
I20240501 07:59:10.853865 6686 tablet_service.cc:1514] Processing DeleteTablet for tablet adf207c019994dd49671bab4145fc268 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:10 UTC) from {username='slave'} at 127.0.0.1:38524
I20240501 07:59:10.853968 6687 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b680bccf110443e9495942863364ff9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:10 UTC) from {username='slave'} at 127.0.0.1:38524
I20240501 07:59:10.854027 6511 leader_election.cc:302] T dbbae25711d94ed29223004d019ef377 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b164cbbc00f4e8eb3254ddd349a3ff9, a505e7cb133c4392bfda8a4a20874ae9; no voters:
I20240501 07:59:10.854097 10310 tablet_replica.cc:330] T adf207c019994dd49671bab4145fc268 P 1f9cb72be98941b5a8e504cbe34132b9: stopping tablet replica
I20240501 07:59:10.854207 10310 raft_consensus.cc:2229] T adf207c019994dd49671bab4145fc268 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:10.854323 10310 raft_consensus.cc:2258] T adf207c019994dd49671bab4145fc268 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:10.854431 6555 tablet_service.cc:1514] Processing DeleteTablet for tablet bab34f083c8045e4a6df373db5ebd54b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:10 UTC) from {username='slave'} at 127.0.0.1:53874
I20240501 07:59:10.855144 10046 raft_consensus.cc:2781] T dbbae25711d94ed29223004d019ef377 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:10.855238 10046 raft_consensus.cc:686] T dbbae25711d94ed29223004d019ef377 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 LEADER]: Becoming Leader. State: Replica: 3b164cbbc00f4e8eb3254ddd349a3ff9, State: Running, Role: LEADER
I20240501 07:59:10.855360 10046 consensus_queue.cc:237] T dbbae25711d94ed29223004d019ef377 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } }
I20240501 07:59:10.855712 10310 ts_tablet_manager.cc:1875] T adf207c019994dd49671bab4145fc268 P 1f9cb72be98941b5a8e504cbe34132b9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:59:10.856027 6556 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b680bccf110443e9495942863364ff9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:10 UTC) from {username='slave'} at 127.0.0.1:53874
I20240501 07:59:10.856779 8904 tablet_service.cc:1514] Processing DeleteTablet for tablet bab34f083c8045e4a6df373db5ebd54b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:10 UTC) from {username='slave'} at 127.0.0.1:47202
I20240501 07:59:10.856994 10466 tablet_replica.cc:330] T bab34f083c8045e4a6df373db5ebd54b P 13d05ebdca8c4f489c597ce550dd773d: stopping tablet replica
I20240501 07:59:10.857128 10466 raft_consensus.cc:2229] T bab34f083c8045e4a6df373db5ebd54b P 13d05ebdca8c4f489c597ce550dd773d [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:10.857252 10466 raft_consensus.cc:2258] T bab34f083c8045e4a6df373db5ebd54b P 13d05ebdca8c4f489c597ce550dd773d [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:10.857771 10466 ts_tablet_manager.cc:1875] T bab34f083c8045e4a6df373db5ebd54b P 13d05ebdca8c4f489c597ce550dd773d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:59:10.857887 10310 ts_tablet_manager.cc:1888] T adf207c019994dd49671bab4145fc268 P 1f9cb72be98941b5a8e504cbe34132b9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:59:10.857946 10310 log.cc:1196] T adf207c019994dd49671bab4145fc268 P 1f9cb72be98941b5a8e504cbe34132b9: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/adf207c019994dd49671bab4145fc268
I20240501 07:59:10.858193 10310 ts_tablet_manager.cc:1909] T adf207c019994dd49671bab4145fc268 P 1f9cb72be98941b5a8e504cbe34132b9: Deleting consensus metadata
I20240501 07:59:10.859871 10466 ts_tablet_manager.cc:1888] T bab34f083c8045e4a6df373db5ebd54b P 13d05ebdca8c4f489c597ce550dd773d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20240501 07:59:10.859938 10466 log.cc:1196] T bab34f083c8045e4a6df373db5ebd54b P 13d05ebdca8c4f489c597ce550dd773d: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/bab34f083c8045e4a6df373db5ebd54b
I20240501 07:59:10.860214 10466 ts_tablet_manager.cc:1909] T bab34f083c8045e4a6df373db5ebd54b P 13d05ebdca8c4f489c597ce550dd773d: Deleting consensus metadata
I20240501 07:59:10.860661 6159 catalog_manager.cc:4830] TS 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132:36773): tablet adf207c019994dd49671bab4145fc268 (table rebalance_test_table_12 [id=f041050203204d0690b4b8f368f834b2]) successfully deleted
I20240501 07:59:10.860814 6159 catalog_manager.cc:4830] TS 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134:40141): tablet bab34f083c8045e4a6df373db5ebd54b (table rebalance_test_table_12 [id=f041050203204d0690b4b8f368f834b2]) successfully deleted
I20240501 07:59:10.861079 10310 tablet_replica.cc:330] T 0b680bccf110443e9495942863364ff9 P 1f9cb72be98941b5a8e504cbe34132b9: stopping tablet replica
I20240501 07:59:10.861333 10310 raft_consensus.cc:2229] T 0b680bccf110443e9495942863364ff9 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:10.861709 10310 raft_consensus.cc:2258] T 0b680bccf110443e9495942863364ff9 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:10.862016 8924 raft_consensus.cc:1261] T ff9dc384dbac47dc9795617c64688943 P 13d05ebdca8c4f489c597ce550dd773d [term 2 FOLLOWER]: Refusing update from remote peer 1f9cb72be98941b5a8e504cbe34132b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 07:59:10.862100 6168 catalog_manager.cc:5484] T dbbae25711d94ed29223004d019ef377 P 3b164cbbc00f4e8eb3254ddd349a3ff9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } health_report { overall_health: UNKNOWN } } }
I20240501 07:59:10.862412 10288 consensus_queue.cc:1035] T ff9dc384dbac47dc9795617c64688943 P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 07:59:10.863095 10310 ts_tablet_manager.cc:1875] T 0b680bccf110443e9495942863364ff9 P 1f9cb72be98941b5a8e504cbe34132b9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:59:10.865628 10753 tablet_replica.cc:330] T bab34f083c8045e4a6df373db5ebd54b P 3b164cbbc00f4e8eb3254ddd349a3ff9: stopping tablet replica
I20240501 07:59:10.865860 10753 raft_consensus.cc:2229] T bab34f083c8045e4a6df373db5ebd54b P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:10.866072 10753 raft_consensus.cc:2258] T bab34f083c8045e4a6df373db5ebd54b P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:10.866677 10753 ts_tablet_manager.cc:1875] T bab34f083c8045e4a6df373db5ebd54b P 3b164cbbc00f4e8eb3254ddd349a3ff9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:59:10.868041 9035 tablet_service.cc:1514] Processing DeleteTablet for tablet adf207c019994dd49671bab4145fc268 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:10 UTC) from {username='slave'} at 127.0.0.1:44236
I20240501 07:59:10.868253 10532 tablet_replica.cc:330] T adf207c019994dd49671bab4145fc268 P a505e7cb133c4392bfda8a4a20874ae9: stopping tablet replica
I20240501 07:59:10.868387 10309 tablet_replica.cc:330] T 0b680bccf110443e9495942863364ff9 P 8b8b3ad3aa8f4053b2167dd550f2086f: stopping tablet replica
I20240501 07:59:10.868386 10532 raft_consensus.cc:2229] T adf207c019994dd49671bab4145fc268 P a505e7cb133c4392bfda8a4a20874ae9 [term 1 LEARNER]: Raft consensus shutting down.
I20240501 07:59:10.868490 10532 raft_consensus.cc:2258] T adf207c019994dd49671bab4145fc268 P a505e7cb133c4392bfda8a4a20874ae9 [term 1 LEARNER]: Raft consensus is shut down!
I20240501 07:59:10.868547 10309 raft_consensus.cc:2229] T 0b680bccf110443e9495942863364ff9 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:10.868685 10309 raft_consensus.cc:2258] T 0b680bccf110443e9495942863364ff9 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:10.869158 6157 catalog_manager.cc:4830] TS 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129:46875): tablet adf207c019994dd49671bab4145fc268 (table rebalance_test_table_12 [id=f041050203204d0690b4b8f368f834b2]) successfully deleted
I20240501 07:59:10.869711 10309 ts_tablet_manager.cc:1875] T 0b680bccf110443e9495942863364ff9 P 8b8b3ad3aa8f4053b2167dd550f2086f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:59:10.869920 10532 ts_tablet_manager.cc:1875] T adf207c019994dd49671bab4145fc268 P a505e7cb133c4392bfda8a4a20874ae9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:59:10.871807 10309 ts_tablet_manager.cc:1888] T 0b680bccf110443e9495942863364ff9 P 8b8b3ad3aa8f4053b2167dd550f2086f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:59:10.871872 10309 log.cc:1196] T 0b680bccf110443e9495942863364ff9 P 8b8b3ad3aa8f4053b2167dd550f2086f: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/0b680bccf110443e9495942863364ff9
I20240501 07:59:10.872047 10532 ts_tablet_manager.cc:1888] T adf207c019994dd49671bab4145fc268 P a505e7cb133c4392bfda8a4a20874ae9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:59:10.872113 10532 log.cc:1196] T adf207c019994dd49671bab4145fc268 P a505e7cb133c4392bfda8a4a20874ae9: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/adf207c019994dd49671bab4145fc268
I20240501 07:59:10.872121 10309 ts_tablet_manager.cc:1909] T 0b680bccf110443e9495942863364ff9 P 8b8b3ad3aa8f4053b2167dd550f2086f: Deleting consensus metadata
I20240501 07:59:10.872376 10532 ts_tablet_manager.cc:1909] T adf207c019994dd49671bab4145fc268 P a505e7cb133c4392bfda8a4a20874ae9: Deleting consensus metadata
I20240501 07:59:10.872385 8793 raft_consensus.cc:1261] T c3a9036ccfee45498bca9681518d3699 P 74077b61d6f54342bf9736b4a9f2eade [term 2 FOLLOWER]: Refusing update from remote peer 8315d22c56f7439a9e1fc82f153f018c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 07:59:10.872829 9489 consensus_queue.cc:1035] T c3a9036ccfee45498bca9681518d3699 P 8315d22c56f7439a9e1fc82f153f018c [LEADER]: Connected to new peer: Peer: permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 07:59:10.874938 10309 tablet_replica.cc:330] T bab34f083c8045e4a6df373db5ebd54b P 8b8b3ad3aa8f4053b2167dd550f2086f: stopping tablet replica
I20240501 07:59:10.875082 10309 raft_consensus.cc:2229] T bab34f083c8045e4a6df373db5ebd54b P 8b8b3ad3aa8f4053b2167dd550f2086f [term 2 LEADER]: Raft consensus shutting down.
I20240501 07:59:10.875308 10309 pending_rounds.cc:62] T bab34f083c8045e4a6df373db5ebd54b P 8b8b3ad3aa8f4053b2167dd550f2086f: Trying to abort 1 pending ops.
I20240501 07:59:10.875352 10309 pending_rounds.cc:68] T bab34f083c8045e4a6df373db5ebd54b P 8b8b3ad3aa8f4053b2167dd550f2086f: Aborting op as it isn't in flight: id { term: 2 index: 6 } timestamp: 7022798236638842880 op_type: NO_OP noop_request { }
I20240501 07:59:10.875419 10309 raft_consensus.cc:2866] T bab34f083c8045e4a6df373db5ebd54b P 8b8b3ad3aa8f4053b2167dd550f2086f [term 2 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240501 07:59:10.875389 6157 catalog_manager.cc:4830] TS 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129:46875): tablet 0b680bccf110443e9495942863364ff9 (table rebalance_test_table_12 [id=f041050203204d0690b4b8f368f834b2]) successfully deleted
I20240501 07:59:10.875470 10309 raft_consensus.cc:2258] T bab34f083c8045e4a6df373db5ebd54b P 8b8b3ad3aa8f4053b2167dd550f2086f [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:10.875845 10753 ts_tablet_manager.cc:1888] T bab34f083c8045e4a6df373db5ebd54b P 3b164cbbc00f4e8eb3254ddd349a3ff9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20240501 07:59:10.875968 10753 log.cc:1196] T bab34f083c8045e4a6df373db5ebd54b P 3b164cbbc00f4e8eb3254ddd349a3ff9: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/bab34f083c8045e4a6df373db5ebd54b
I20240501 07:59:10.876590 6158 catalog_manager.cc:4830] TS a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): tablet adf207c019994dd49671bab4145fc268 (table rebalance_test_table_12 [id=f041050203204d0690b4b8f368f834b2]) successfully deleted
I20240501 07:59:10.876811 10753 ts_tablet_manager.cc:1909] T bab34f083c8045e4a6df373db5ebd54b P 3b164cbbc00f4e8eb3254ddd349a3ff9: Deleting consensus metadata
I20240501 07:59:10.877529 10310 ts_tablet_manager.cc:1888] T 0b680bccf110443e9495942863364ff9 P 1f9cb72be98941b5a8e504cbe34132b9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:59:10.877611 10310 log.cc:1196] T 0b680bccf110443e9495942863364ff9 P 1f9cb72be98941b5a8e504cbe34132b9: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/0b680bccf110443e9495942863364ff9
I20240501 07:59:10.877880 10310 ts_tablet_manager.cc:1909] T 0b680bccf110443e9495942863364ff9 P 1f9cb72be98941b5a8e504cbe34132b9: Deleting consensus metadata
I20240501 07:59:10.878521 10753 tablet_replica.cc:330] T 0b680bccf110443e9495942863364ff9 P 3b164cbbc00f4e8eb3254ddd349a3ff9: stopping tablet replica
I20240501 07:59:10.878717 10753 raft_consensus.cc:2229] T 0b680bccf110443e9495942863364ff9 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 LEADER]: Raft consensus shutting down.
I20240501 07:59:10.878966 6160 catalog_manager.cc:4830] TS 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131:34471): tablet bab34f083c8045e4a6df373db5ebd54b (table rebalance_test_table_12 [id=f041050203204d0690b4b8f368f834b2]) successfully deleted
I20240501 07:59:10.879298 10753 raft_consensus.cc:2258] T 0b680bccf110443e9495942863364ff9 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:10.884124 10309 ts_tablet_manager.cc:1875] T bab34f083c8045e4a6df373db5ebd54b P 8b8b3ad3aa8f4053b2167dd550f2086f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:59:10.885707 10753 ts_tablet_manager.cc:1875] T 0b680bccf110443e9495942863364ff9 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:59:10.886426 6159 catalog_manager.cc:4830] TS 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132:36773): tablet 0b680bccf110443e9495942863364ff9 (table rebalance_test_table_12 [id=f041050203204d0690b4b8f368f834b2]) successfully deleted
I20240501 07:59:10.887956 10753 ts_tablet_manager.cc:1888] T 0b680bccf110443e9495942863364ff9 P 3b164cbbc00f4e8eb3254ddd349a3ff9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:59:10.888041 10753 log.cc:1196] T 0b680bccf110443e9495942863364ff9 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/0b680bccf110443e9495942863364ff9
I20240501 07:59:10.888341 10753 ts_tablet_manager.cc:1909] T 0b680bccf110443e9495942863364ff9 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Deleting consensus metadata
I20240501 07:59:10.889174 6160 catalog_manager.cc:4830] TS 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131:34471): tablet 0b680bccf110443e9495942863364ff9 (table rebalance_test_table_12 [id=f041050203204d0690b4b8f368f834b2]) successfully deleted
I20240501 07:59:10.891261 6577 raft_consensus.cc:1261] T ff9dc384dbac47dc9795617c64688943 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 2 FOLLOWER]: Refusing update from remote peer 1f9cb72be98941b5a8e504cbe34132b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 07:59:10.892218 10288 consensus_queue.cc:1035] T ff9dc384dbac47dc9795617c64688943 P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 07:59:10.892293 10309 ts_tablet_manager.cc:1888] T bab34f083c8045e4a6df373db5ebd54b P 8b8b3ad3aa8f4053b2167dd550f2086f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.6
I20240501 07:59:10.892361 10309 log.cc:1196] T bab34f083c8045e4a6df373db5ebd54b P 8b8b3ad3aa8f4053b2167dd550f2086f: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/bab34f083c8045e4a6df373db5ebd54b
I20240501 07:59:10.892624 10309 ts_tablet_manager.cc:1909] T bab34f083c8045e4a6df373db5ebd54b P 8b8b3ad3aa8f4053b2167dd550f2086f: Deleting consensus metadata
I20240501 07:59:10.893306 6157 catalog_manager.cc:4830] TS 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129:46875): tablet bab34f083c8045e4a6df373db5ebd54b (table rebalance_test_table_12 [id=f041050203204d0690b4b8f368f834b2]) successfully deleted
I20240501 07:59:10.893684 10751 ts_tablet_manager.cc:1888] T adf207c019994dd49671bab4145fc268 P 8315d22c56f7439a9e1fc82f153f018c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:59:10.893759 10751 log.cc:1196] T adf207c019994dd49671bab4145fc268 P 8315d22c56f7439a9e1fc82f153f018c: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/adf207c019994dd49671bab4145fc268
I20240501 07:59:10.894016 10751 ts_tablet_manager.cc:1909] T adf207c019994dd49671bab4145fc268 P 8315d22c56f7439a9e1fc82f153f018c: Deleting consensus metadata
I20240501 07:59:10.894713 6159 catalog_manager.cc:4830] TS 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130:32941): tablet adf207c019994dd49671bab4145fc268 (table rebalance_test_table_12 [id=f041050203204d0690b4b8f368f834b2]) successfully deleted
W20240501 07:59:10.895797 6157 catalog_manager.cc:4797] TS 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133:39859): delete failed for tablet 0b680bccf110443e9495942863364ff9 because the tablet was not found. No further retry: Not found: Tablet not found: 0b680bccf110443e9495942863364ff9
I20240501 07:59:10.897658 10751 tablet_replica.cc:330] T bab34f083c8045e4a6df373db5ebd54b P 8315d22c56f7439a9e1fc82f153f018c: stopping tablet replica
I20240501 07:59:10.897903 10751 raft_consensus.cc:2229] T bab34f083c8045e4a6df373db5ebd54b P 8315d22c56f7439a9e1fc82f153f018c [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:10.898061 10751 raft_consensus.cc:2258] T bab34f083c8045e4a6df373db5ebd54b P 8315d22c56f7439a9e1fc82f153f018c [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:10.898663 10751 ts_tablet_manager.cc:1875] T bab34f083c8045e4a6df373db5ebd54b P 8315d22c56f7439a9e1fc82f153f018c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:59:10.901638 10751 ts_tablet_manager.cc:1888] T bab34f083c8045e4a6df373db5ebd54b P 8315d22c56f7439a9e1fc82f153f018c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20240501 07:59:10.901762 10751 log.cc:1196] T bab34f083c8045e4a6df373db5ebd54b P 8315d22c56f7439a9e1fc82f153f018c: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/bab34f083c8045e4a6df373db5ebd54b
I20240501 07:59:10.902065 10751 ts_tablet_manager.cc:1909] T bab34f083c8045e4a6df373db5ebd54b P 8315d22c56f7439a9e1fc82f153f018c: Deleting consensus metadata
I20240501 07:59:10.902757 6159 catalog_manager.cc:4830] TS 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130:32941): tablet bab34f083c8045e4a6df373db5ebd54b (table rebalance_test_table_12 [id=f041050203204d0690b4b8f368f834b2]) successfully deleted
I20240501 07:59:10.911993 10717 tablet_bootstrap.cc:492] T 2398430a23094711a13a86cc88e27a78 P 74077b61d6f54342bf9736b4a9f2eade: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:10.912353 10743 tablet_bootstrap.cc:492] T 72aa4dc7664b48b7a73d34feba7d44d3 P 74077b61d6f54342bf9736b4a9f2eade: Bootstrap starting.
I20240501 07:59:10.912422 10717 tablet_bootstrap.cc:492] T 2398430a23094711a13a86cc88e27a78 P 74077b61d6f54342bf9736b4a9f2eade: Bootstrap complete.
I20240501 07:59:10.912823 10717 ts_tablet_manager.cc:1367] T 2398430a23094711a13a86cc88e27a78 P 74077b61d6f54342bf9736b4a9f2eade: Time spent bootstrapping tablet: real 0.077s user 0.029s sys 0.000s
I20240501 07:59:10.913422 10717 raft_consensus.cc:348] T 2398430a23094711a13a86cc88e27a78 P 74077b61d6f54342bf9736b4a9f2eade [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: true } }
I20240501 07:59:10.913578 10717 raft_consensus.cc:729] T 2398430a23094711a13a86cc88e27a78 P 74077b61d6f54342bf9736b4a9f2eade [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 74077b61d6f54342bf9736b4a9f2eade, State: Initialized, Role: LEARNER
I20240501 07:59:10.913710 10717 consensus_queue.cc:260] T 2398430a23094711a13a86cc88e27a78 P 74077b61d6f54342bf9736b4a9f2eade [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: true } }
I20240501 07:59:10.914023 10717 ts_tablet_manager.cc:1398] T 2398430a23094711a13a86cc88e27a78 P 74077b61d6f54342bf9736b4a9f2eade: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:10.915004 6335 tablet_copy_service.cc:342] P 8b8b3ad3aa8f4053b2167dd550f2086f: Request end of tablet copy session 74077b61d6f54342bf9736b4a9f2eade-2398430a23094711a13a86cc88e27a78 received from {username='slave'} at 127.0.99.133:50111
I20240501 07:59:10.915076 10742 tablet_bootstrap.cc:492] T ab1666f1e5204416adb435d9bbe80433 P 8b8b3ad3aa8f4053b2167dd550f2086f: Bootstrap starting.
I20240501 07:59:10.915096 6335 tablet_copy_service.cc:433] P 8b8b3ad3aa8f4053b2167dd550f2086f: ending tablet copy session 74077b61d6f54342bf9736b4a9f2eade-2398430a23094711a13a86cc88e27a78 on tablet 2398430a23094711a13a86cc88e27a78 with peer 74077b61d6f54342bf9736b4a9f2eade
I20240501 07:59:10.916191 10742 tablet_bootstrap.cc:654] T ab1666f1e5204416adb435d9bbe80433 P 8b8b3ad3aa8f4053b2167dd550f2086f: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:10.913717 10743 tablet_bootstrap.cc:654] T 72aa4dc7664b48b7a73d34feba7d44d3 P 74077b61d6f54342bf9736b4a9f2eade: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:10.918550 10744 tablet_bootstrap.cc:492] T 542c9ac08f7d4bec9b15a309c2e21364 P 13d05ebdca8c4f489c597ce550dd773d: No bootstrap required, opened a new log
I20240501 07:59:10.918567 10589 tablet_bootstrap.cc:492] T ab1666f1e5204416adb435d9bbe80433 P 83f20e84af4c40d09a4804b6b26e697f: No bootstrap required, opened a new log
I20240501 07:59:10.918658 10744 ts_tablet_manager.cc:1367] T 542c9ac08f7d4bec9b15a309c2e21364 P 13d05ebdca8c4f489c597ce550dd773d: Time spent bootstrapping tablet: real 0.085s user 0.002s sys 0.000s
I20240501 07:59:10.918668 10589 ts_tablet_manager.cc:1367] T ab1666f1e5204416adb435d9bbe80433 P 83f20e84af4c40d09a4804b6b26e697f: Time spent bootstrapping tablet: real 0.083s user 0.002s sys 0.000s
I20240501 07:59:10.919078 10589 raft_consensus.cc:348] T ab1666f1e5204416adb435d9bbe80433 P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } }
I20240501 07:59:10.919083 10744 raft_consensus.cc:348] T 542c9ac08f7d4bec9b15a309c2e21364 P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } }
I20240501 07:59:10.919184 10589 raft_consensus.cc:374] T ab1666f1e5204416adb435d9bbe80433 P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:10.919184 10744 raft_consensus.cc:374] T 542c9ac08f7d4bec9b15a309c2e21364 P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:10.919216 10589 raft_consensus.cc:729] T ab1666f1e5204416adb435d9bbe80433 P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83f20e84af4c40d09a4804b6b26e697f, State: Initialized, Role: FOLLOWER
I20240501 07:59:10.919229 10744 raft_consensus.cc:729] T 542c9ac08f7d4bec9b15a309c2e21364 P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13d05ebdca8c4f489c597ce550dd773d, State: Initialized, Role: FOLLOWER
I20240501 07:59:10.919350 10589 consensus_queue.cc:260] T ab1666f1e5204416adb435d9bbe80433 P 83f20e84af4c40d09a4804b6b26e697f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } }
I20240501 07:59:10.919350 10744 consensus_queue.cc:260] T 542c9ac08f7d4bec9b15a309c2e21364 P 13d05ebdca8c4f489c597ce550dd773d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } }
I20240501 07:59:10.919620 10589 ts_tablet_manager.cc:1398] T ab1666f1e5204416adb435d9bbe80433 P 83f20e84af4c40d09a4804b6b26e697f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:10.919620 10744 ts_tablet_manager.cc:1398] T 542c9ac08f7d4bec9b15a309c2e21364 P 13d05ebdca8c4f489c597ce550dd773d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:10.919788 10744 tablet_bootstrap.cc:492] T 72aa4dc7664b48b7a73d34feba7d44d3 P 13d05ebdca8c4f489c597ce550dd773d: Bootstrap starting.
I20240501 07:59:10.920732 10744 tablet_bootstrap.cc:654] T 72aa4dc7664b48b7a73d34feba7d44d3 P 13d05ebdca8c4f489c597ce550dd773d: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:10.922104 10744 tablet_bootstrap.cc:492] T 72aa4dc7664b48b7a73d34feba7d44d3 P 13d05ebdca8c4f489c597ce550dd773d: No bootstrap required, opened a new log
I20240501 07:59:10.922191 10744 ts_tablet_manager.cc:1367] T 72aa4dc7664b48b7a73d34feba7d44d3 P 13d05ebdca8c4f489c597ce550dd773d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 07:59:10.922638 10744 raft_consensus.cc:348] T 72aa4dc7664b48b7a73d34feba7d44d3 P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } }
I20240501 07:59:10.922742 10744 raft_consensus.cc:374] T 72aa4dc7664b48b7a73d34feba7d44d3 P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:10.922776 10744 raft_consensus.cc:729] T 72aa4dc7664b48b7a73d34feba7d44d3 P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13d05ebdca8c4f489c597ce550dd773d, State: Initialized, Role: FOLLOWER
I20240501 07:59:10.922896 10744 consensus_queue.cc:260] T 72aa4dc7664b48b7a73d34feba7d44d3 P 13d05ebdca8c4f489c597ce550dd773d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } }
I20240501 07:59:10.923048 10743 tablet_bootstrap.cc:492] T 72aa4dc7664b48b7a73d34feba7d44d3 P 74077b61d6f54342bf9736b4a9f2eade: No bootstrap required, opened a new log
I20240501 07:59:10.923143 10743 ts_tablet_manager.cc:1367] T 72aa4dc7664b48b7a73d34feba7d44d3 P 74077b61d6f54342bf9736b4a9f2eade: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240501 07:59:10.923158 10744 ts_tablet_manager.cc:1398] T 72aa4dc7664b48b7a73d34feba7d44d3 P 13d05ebdca8c4f489c597ce550dd773d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:10.923866 10743 raft_consensus.cc:348] T 72aa4dc7664b48b7a73d34feba7d44d3 P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } }
I20240501 07:59:10.923985 10743 raft_consensus.cc:374] T 72aa4dc7664b48b7a73d34feba7d44d3 P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:10.924027 10743 raft_consensus.cc:729] T 72aa4dc7664b48b7a73d34feba7d44d3 P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74077b61d6f54342bf9736b4a9f2eade, State: Initialized, Role: FOLLOWER
I20240501 07:59:10.924170 10743 consensus_queue.cc:260] T 72aa4dc7664b48b7a73d34feba7d44d3 P 74077b61d6f54342bf9736b4a9f2eade [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } }
I20240501 07:59:10.924677 8680 raft_consensus.cc:1050] T 3745acee2b5543b697a3c8f45d2618bb P 3b164cbbc00f4e8eb3254ddd349a3ff9: attempting to promote NON_VOTER a505e7cb133c4392bfda8a4a20874ae9 to VOTER
I20240501 07:59:10.924989 10743 ts_tablet_manager.cc:1398] T 72aa4dc7664b48b7a73d34feba7d44d3 P 74077b61d6f54342bf9736b4a9f2eade: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:59:10.925143 10743 tablet_bootstrap.cc:492] T ab1666f1e5204416adb435d9bbe80433 P 74077b61d6f54342bf9736b4a9f2eade: Bootstrap starting.
I20240501 07:59:10.925129 8680 consensus_queue.cc:237] T 3745acee2b5543b697a3c8f45d2618bb P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false } }
I20240501 07:59:10.926374 10743 tablet_bootstrap.cc:654] T ab1666f1e5204416adb435d9bbe80433 P 74077b61d6f54342bf9736b4a9f2eade: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:10.927083 6313 raft_consensus.cc:1261] T 3745acee2b5543b697a3c8f45d2618bb P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Refusing update from remote peer 3b164cbbc00f4e8eb3254ddd349a3ff9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:59:10.927407 6311 raft_consensus.cc:1261] T ff9dc384dbac47dc9795617c64688943 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 2 FOLLOWER]: Refusing update from remote peer 1f9cb72be98941b5a8e504cbe34132b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 07:59:10.927793 10288 consensus_queue.cc:1035] T ff9dc384dbac47dc9795617c64688943 P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 07:59:10.927893 9507 consensus_queue.cc:1035] T 3745acee2b5543b697a3c8f45d2618bb P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:59:10.928586 6444 raft_consensus.cc:1261] T 3745acee2b5543b697a3c8f45d2618bb P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Refusing update from remote peer 3b164cbbc00f4e8eb3254ddd349a3ff9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:59:10.929522 9055 raft_consensus.cc:1261] T 3745acee2b5543b697a3c8f45d2618bb P a505e7cb133c4392bfda8a4a20874ae9 [term 1 LEARNER]: Refusing update from remote peer 3b164cbbc00f4e8eb3254ddd349a3ff9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:59:10.929708 10237 consensus_queue.cc:1035] T 3745acee2b5543b697a3c8f45d2618bb P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:59:10.929926 9507 consensus_queue.cc:1035] T 3745acee2b5543b697a3c8f45d2618bb P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:59:10.937422 10592 tablet_bootstrap.cc:492] T 72aa4dc7664b48b7a73d34feba7d44d3 P a505e7cb133c4392bfda8a4a20874ae9: No bootstrap required, opened a new log
I20240501 07:59:10.937547 10592 ts_tablet_manager.cc:1367] T 72aa4dc7664b48b7a73d34feba7d44d3 P a505e7cb133c4392bfda8a4a20874ae9: Time spent bootstrapping tablet: real 0.099s user 0.000s sys 0.002s
I20240501 07:59:10.938055 10592 raft_consensus.cc:348] T 72aa4dc7664b48b7a73d34feba7d44d3 P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } }
I20240501 07:59:10.938191 10592 raft_consensus.cc:374] T 72aa4dc7664b48b7a73d34feba7d44d3 P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:10.938243 10592 raft_consensus.cc:729] T 72aa4dc7664b48b7a73d34feba7d44d3 P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a505e7cb133c4392bfda8a4a20874ae9, State: Initialized, Role: FOLLOWER
I20240501 07:59:10.938364 10592 consensus_queue.cc:260] T 72aa4dc7664b48b7a73d34feba7d44d3 P a505e7cb133c4392bfda8a4a20874ae9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } }
I20240501 07:59:10.938680 10592 ts_tablet_manager.cc:1398] T 72aa4dc7664b48b7a73d34feba7d44d3 P a505e7cb133c4392bfda8a4a20874ae9: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 07:59:10.938871 10592 tablet_bootstrap.cc:492] T 542c9ac08f7d4bec9b15a309c2e21364 P a505e7cb133c4392bfda8a4a20874ae9: Bootstrap starting.
I20240501 07:59:10.939823 10592 tablet_bootstrap.cc:654] T 542c9ac08f7d4bec9b15a309c2e21364 P a505e7cb133c4392bfda8a4a20874ae9: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:10.940834 10742 tablet_bootstrap.cc:492] T ab1666f1e5204416adb435d9bbe80433 P 8b8b3ad3aa8f4053b2167dd550f2086f: No bootstrap required, opened a new log
I20240501 07:59:10.940858 10592 tablet_bootstrap.cc:492] T 542c9ac08f7d4bec9b15a309c2e21364 P a505e7cb133c4392bfda8a4a20874ae9: No bootstrap required, opened a new log
I20240501 07:59:10.940935 10592 ts_tablet_manager.cc:1367] T 542c9ac08f7d4bec9b15a309c2e21364 P a505e7cb133c4392bfda8a4a20874ae9: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240501 07:59:10.940946 10742 ts_tablet_manager.cc:1367] T ab1666f1e5204416adb435d9bbe80433 P 8b8b3ad3aa8f4053b2167dd550f2086f: Time spent bootstrapping tablet: real 0.026s user 0.001s sys 0.000s
I20240501 07:59:10.941375 10352 ts_tablet_manager.cc:902] T 6c3c0d6fe41f4ae196ae29beb9b68125 P a505e7cb133c4392bfda8a4a20874ae9: Initiating tablet copy from peer 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129:46875)
I20240501 07:59:10.941438 10742 raft_consensus.cc:348] T ab1666f1e5204416adb435d9bbe80433 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } }
I20240501 07:59:10.941411 10592 raft_consensus.cc:348] T 542c9ac08f7d4bec9b15a309c2e21364 P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } }
I20240501 07:59:10.941551 10742 raft_consensus.cc:374] T ab1666f1e5204416adb435d9bbe80433 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:10.941555 10592 raft_consensus.cc:374] T 542c9ac08f7d4bec9b15a309c2e21364 P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:10.941591 10592 raft_consensus.cc:729] T 542c9ac08f7d4bec9b15a309c2e21364 P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a505e7cb133c4392bfda8a4a20874ae9, State: Initialized, Role: FOLLOWER
I20240501 07:59:10.941587 10742 raft_consensus.cc:729] T ab1666f1e5204416adb435d9bbe80433 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b8b3ad3aa8f4053b2167dd550f2086f, State: Initialized, Role: FOLLOWER
I20240501 07:59:10.941722 10592 consensus_queue.cc:260] T 542c9ac08f7d4bec9b15a309c2e21364 P a505e7cb133c4392bfda8a4a20874ae9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } }
I20240501 07:59:10.941725 10742 consensus_queue.cc:260] T ab1666f1e5204416adb435d9bbe80433 P 8b8b3ad3aa8f4053b2167dd550f2086f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } }
I20240501 07:59:10.941859 10352 tablet_copy_client.cc:308] T 6c3c0d6fe41f4ae196ae29beb9b68125 P a505e7cb133c4392bfda8a4a20874ae9: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:46875
I20240501 07:59:10.941991 10592 ts_tablet_manager.cc:1398] T 542c9ac08f7d4bec9b15a309c2e21364 P a505e7cb133c4392bfda8a4a20874ae9: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:59:10.943248 10742 ts_tablet_manager.cc:1398] T ab1666f1e5204416adb435d9bbe80433 P 8b8b3ad3aa8f4053b2167dd550f2086f: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 07:59:10.946599 9507 raft_consensus.cc:2932] T 3745acee2b5543b697a3c8f45d2618bb P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false } } }
I20240501 07:59:10.949146 9400 raft_consensus.cc:482] T 542c9ac08f7d4bec9b15a309c2e21364 P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:10.949258 9400 raft_consensus.cc:504] T 542c9ac08f7d4bec9b15a309c2e21364 P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } }
I20240501 07:59:10.949303 9055 raft_consensus.cc:2932] T 3745acee2b5543b697a3c8f45d2618bb P a505e7cb133c4392bfda8a4a20874ae9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false } } }
I20240501 07:59:10.949759 9400 leader_election.cc:288] T 542c9ac08f7d4bec9b15a309c2e21364 P 13d05ebdca8c4f489c597ce550dd773d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607), 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132:36773)
I20240501 07:59:10.950412 9056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "542c9ac08f7d4bec9b15a309c2e21364" candidate_uuid: "13d05ebdca8c4f489c597ce550dd773d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a505e7cb133c4392bfda8a4a20874ae9" is_pre_election: true
I20240501 07:59:10.950542 9056 raft_consensus.cc:2454] T 542c9ac08f7d4bec9b15a309c2e21364 P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 13d05ebdca8c4f489c597ce550dd773d in term 0.
I20240501 07:59:10.950750 10743 tablet_bootstrap.cc:492] T ab1666f1e5204416adb435d9bbe80433 P 74077b61d6f54342bf9736b4a9f2eade: No bootstrap required, opened a new log
I20240501 07:59:10.950863 10743 ts_tablet_manager.cc:1367] T ab1666f1e5204416adb435d9bbe80433 P 74077b61d6f54342bf9736b4a9f2eade: Time spent bootstrapping tablet: real 0.026s user 0.002s sys 0.000s
I20240501 07:59:10.950973 6708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "542c9ac08f7d4bec9b15a309c2e21364" candidate_uuid: "13d05ebdca8c4f489c597ce550dd773d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f9cb72be98941b5a8e504cbe34132b9" is_pre_election: true
I20240501 07:59:10.951124 6708 raft_consensus.cc:2454] T 542c9ac08f7d4bec9b15a309c2e21364 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 13d05ebdca8c4f489c597ce550dd773d in term 0.
I20240501 07:59:10.951439 10743 raft_consensus.cc:348] T ab1666f1e5204416adb435d9bbe80433 P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } }
I20240501 07:59:10.951495 8860 leader_election.cc:302] T 542c9ac08f7d4bec9b15a309c2e21364 P 13d05ebdca8c4f489c597ce550dd773d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13d05ebdca8c4f489c597ce550dd773d, 1f9cb72be98941b5a8e504cbe34132b9; no voters:
I20240501 07:59:10.951568 10743 raft_consensus.cc:374] T ab1666f1e5204416adb435d9bbe80433 P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:10.951601 10743 raft_consensus.cc:729] T ab1666f1e5204416adb435d9bbe80433 P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74077b61d6f54342bf9736b4a9f2eade, State: Initialized, Role: FOLLOWER
I20240501 07:59:10.951725 10743 consensus_queue.cc:260] T ab1666f1e5204416adb435d9bbe80433 P 74077b61d6f54342bf9736b4a9f2eade [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } }
I20240501 07:59:10.952008 10743 ts_tablet_manager.cc:1398] T ab1666f1e5204416adb435d9bbe80433 P 74077b61d6f54342bf9736b4a9f2eade: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:10.952347 6311 raft_consensus.cc:2932] T 3745acee2b5543b697a3c8f45d2618bb P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false } } }
I20240501 07:59:10.952409 6444 raft_consensus.cc:2932] T 3745acee2b5543b697a3c8f45d2618bb P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false } } }
I20240501 07:59:10.954690 9400 raft_consensus.cc:2781] T 542c9ac08f7d4bec9b15a309c2e21364 P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:10.954756 9400 raft_consensus.cc:482] T 542c9ac08f7d4bec9b15a309c2e21364 P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:10.954793 9400 raft_consensus.cc:3037] T 542c9ac08f7d4bec9b15a309c2e21364 P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:10.955617 6165 catalog_manager.cc:5484] T 3745acee2b5543b697a3c8f45d2618bb P a505e7cb133c4392bfda8a4a20874ae9 reported cstate change: config changed from index 4 to 5, a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false } } }
I20240501 07:59:10.955803 9400 raft_consensus.cc:504] T 542c9ac08f7d4bec9b15a309c2e21364 P 13d05ebdca8c4f489c597ce550dd773d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } }
I20240501 07:59:10.956142 9400 leader_election.cc:288] T 542c9ac08f7d4bec9b15a309c2e21364 P 13d05ebdca8c4f489c597ce550dd773d [CANDIDATE]: Term 1 election: Requested vote from peers a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607), 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132:36773)
I20240501 07:59:10.957149 9056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "542c9ac08f7d4bec9b15a309c2e21364" candidate_uuid: "13d05ebdca8c4f489c597ce550dd773d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a505e7cb133c4392bfda8a4a20874ae9"
I20240501 07:59:10.957170 6708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "542c9ac08f7d4bec9b15a309c2e21364" candidate_uuid: "13d05ebdca8c4f489c597ce550dd773d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f9cb72be98941b5a8e504cbe34132b9"
I20240501 07:59:10.957293 6708 raft_consensus.cc:3037] T 542c9ac08f7d4bec9b15a309c2e21364 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:10.957304 9056 raft_consensus.cc:3037] T 542c9ac08f7d4bec9b15a309c2e21364 P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:10.958302 9056 raft_consensus.cc:2454] T 542c9ac08f7d4bec9b15a309c2e21364 P a505e7cb133c4392bfda8a4a20874ae9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13d05ebdca8c4f489c597ce550dd773d in term 1.
I20240501 07:59:10.958349 6708 raft_consensus.cc:2454] T 542c9ac08f7d4bec9b15a309c2e21364 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13d05ebdca8c4f489c597ce550dd773d in term 1.
I20240501 07:59:10.958774 8860 leader_election.cc:302] T 542c9ac08f7d4bec9b15a309c2e21364 P 13d05ebdca8c4f489c597ce550dd773d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13d05ebdca8c4f489c597ce550dd773d, 1f9cb72be98941b5a8e504cbe34132b9; no voters:
I20240501 07:59:10.959285 9400 raft_consensus.cc:2781] T 542c9ac08f7d4bec9b15a309c2e21364 P 13d05ebdca8c4f489c597ce550dd773d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:10.959372 9400 raft_consensus.cc:686] T 542c9ac08f7d4bec9b15a309c2e21364 P 13d05ebdca8c4f489c597ce550dd773d [term 1 LEADER]: Becoming Leader. State: Replica: 13d05ebdca8c4f489c597ce550dd773d, State: Running, Role: LEADER
I20240501 07:59:10.959506 9400 consensus_queue.cc:237] T 542c9ac08f7d4bec9b15a309c2e21364 P 13d05ebdca8c4f489c597ce550dd773d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } }
I20240501 07:59:10.962922 6171 catalog_manager.cc:5484] T 542c9ac08f7d4bec9b15a309c2e21364 P 13d05ebdca8c4f489c597ce550dd773d reported cstate change: term changed from 0 to 1, leader changed from <none> to 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134). New cstate: current_term: 1 leader_uuid: "13d05ebdca8c4f489c597ce550dd773d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } health_report { overall_health: HEALTHY } } }
I20240501 07:59:10.963064 6335 tablet_copy_service.cc:140] P 8b8b3ad3aa8f4053b2167dd550f2086f: Received BeginTabletCopySession request for tablet 6c3c0d6fe41f4ae196ae29beb9b68125 from peer a505e7cb133c4392bfda8a4a20874ae9 ({username='slave'} at 127.0.99.135:38447)
I20240501 07:59:10.963161 6335 tablet_copy_service.cc:161] P 8b8b3ad3aa8f4053b2167dd550f2086f: Beginning new tablet copy session on tablet 6c3c0d6fe41f4ae196ae29beb9b68125 from peer a505e7cb133c4392bfda8a4a20874ae9 at {username='slave'} at 127.0.99.135:38447: session id = a505e7cb133c4392bfda8a4a20874ae9-6c3c0d6fe41f4ae196ae29beb9b68125
I20240501 07:59:10.963367 6708 consensus_queue.cc:237] T ff9dc384dbac47dc9795617c64688943 P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false } }
I20240501 07:59:10.964184 6335 tablet_copy_source_session.cc:205] T 6c3c0d6fe41f4ae196ae29beb9b68125 P 8b8b3ad3aa8f4053b2167dd550f2086f: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:10.965482 10352 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c3c0d6fe41f4ae196ae29beb9b68125. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:10.965950 8924 raft_consensus.cc:1261] T ff9dc384dbac47dc9795617c64688943 P 13d05ebdca8c4f489c597ce550dd773d [term 2 FOLLOWER]: Refusing update from remote peer 1f9cb72be98941b5a8e504cbe34132b9: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 07:59:10.967084 10324 consensus_queue.cc:1035] T ff9dc384dbac47dc9795617c64688943 P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 07:59:10.967691 6311 raft_consensus.cc:1261] T ff9dc384dbac47dc9795617c64688943 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 2 FOLLOWER]: Refusing update from remote peer 1f9cb72be98941b5a8e504cbe34132b9: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 07:59:10.967810 10352 tablet_copy_client.cc:783] T 6c3c0d6fe41f4ae196ae29beb9b68125 P a505e7cb133c4392bfda8a4a20874ae9: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:10.967962 10352 tablet_copy_client.cc:647] T 6c3c0d6fe41f4ae196ae29beb9b68125 P a505e7cb133c4392bfda8a4a20874ae9: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:10.968755 10324 consensus_queue.cc:1035] T ff9dc384dbac47dc9795617c64688943 P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 07:59:10.968865 6577 consensus_queue.cc:237] T 3745acee2b5543b697a3c8f45d2618bb P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false } }
I20240501 07:59:10.969584 9990 raft_consensus.cc:2932] T ff9dc384dbac47dc9795617c64688943 P 1f9cb72be98941b5a8e504cbe34132b9 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false } } }
I20240501 07:59:10.971321 9056 raft_consensus.cc:1261] T 3745acee2b5543b697a3c8f45d2618bb P a505e7cb133c4392bfda8a4a20874ae9 [term 1 FOLLOWER]: Refusing update from remote peer 3b164cbbc00f4e8eb3254ddd349a3ff9: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 07:59:10.971372 6444 raft_consensus.cc:1261] T 3745acee2b5543b697a3c8f45d2618bb P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Refusing update from remote peer 3b164cbbc00f4e8eb3254ddd349a3ff9: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 07:59:10.972028 6159 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ff9dc384dbac47dc9795617c64688943 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 07:59:10.972256 10046 consensus_queue.cc:1035] T 3745acee2b5543b697a3c8f45d2618bb P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 07:59:10.972206 8924 raft_consensus.cc:2932] T ff9dc384dbac47dc9795617c64688943 P 13d05ebdca8c4f489c597ce550dd773d [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false } } }
I20240501 07:59:10.973119 6708 raft_consensus.cc:1261] T c3a9036ccfee45498bca9681518d3699 P 1f9cb72be98941b5a8e504cbe34132b9 [term 2 FOLLOWER]: Refusing update from remote peer 8315d22c56f7439a9e1fc82f153f018c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 07:59:10.973552 10352 tablet_copy_client.cc:523] T 6c3c0d6fe41f4ae196ae29beb9b68125 P a505e7cb133c4392bfda8a4a20874ae9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:10.973876 9489 consensus_queue.cc:1035] T c3a9036ccfee45498bca9681518d3699 P 8315d22c56f7439a9e1fc82f153f018c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 07:59:10.974108 10046 consensus_queue.cc:1035] T 3745acee2b5543b697a3c8f45d2618bb P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 07:59:10.975108 6311 raft_consensus.cc:2932] T ff9dc384dbac47dc9795617c64688943 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false } } }
I20240501 07:59:10.976020 10046 raft_consensus.cc:2932] T 3745acee2b5543b697a3c8f45d2618bb P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false } } }
I20240501 07:59:10.977424 6577 raft_consensus.cc:1261] T c3a9036ccfee45498bca9681518d3699 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 2 FOLLOWER]: Refusing update from remote peer 8315d22c56f7439a9e1fc82f153f018c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 07:59:10.978291 10352 tablet_bootstrap.cc:492] T 6c3c0d6fe41f4ae196ae29beb9b68125 P a505e7cb133c4392bfda8a4a20874ae9: Bootstrap starting.
I20240501 07:59:10.977991 9056 raft_consensus.cc:2932] T 3745acee2b5543b697a3c8f45d2618bb P a505e7cb133c4392bfda8a4a20874ae9 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false } } }
I20240501 07:59:10.978590 9116 consensus_queue.cc:1035] T c3a9036ccfee45498bca9681518d3699 P 8315d22c56f7439a9e1fc82f153f018c [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 07:59:10.978560 6164 catalog_manager.cc:5484] T ff9dc384dbac47dc9795617c64688943 P 1f9cb72be98941b5a8e504cbe34132b9 reported cstate change: config changed from index 5 to 7, VOTER 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131) evicted. New cstate: current_term: 2 leader_uuid: "1f9cb72be98941b5a8e504cbe34132b9" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 07:59:10.981884 6170 catalog_manager.cc:5484] T 3745acee2b5543b697a3c8f45d2618bb P a505e7cb133c4392bfda8a4a20874ae9 reported cstate change: config changed from index 5 to 6, VOTER 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false } } }
I20240501 07:59:10.983129 6556 tablet_service.cc:1514] Processing DeleteTablet for tablet ff9dc384dbac47dc9795617c64688943 with delete_type TABLET_DATA_TOMBSTONED (TS 3b164cbbc00f4e8eb3254ddd349a3ff9 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:53874
I20240501 07:59:10.983371 10753 tablet_replica.cc:330] T ff9dc384dbac47dc9795617c64688943 P 3b164cbbc00f4e8eb3254ddd349a3ff9: stopping tablet replica
I20240501 07:59:10.983546 10753 raft_consensus.cc:2229] T ff9dc384dbac47dc9795617c64688943 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:10.983654 10753 raft_consensus.cc:2258] T ff9dc384dbac47dc9795617c64688943 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:10.983983 10403 raft_consensus.cc:482] T 72aa4dc7664b48b7a73d34feba7d44d3 P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:10.984074 10403 raft_consensus.cc:504] T 72aa4dc7664b48b7a73d34feba7d44d3 P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } }
I20240501 07:59:10.984548 10403 leader_election.cc:288] T 72aa4dc7664b48b7a73d34feba7d44d3 P 74077b61d6f54342bf9736b4a9f2eade [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134:40141), a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607)
I20240501 07:59:10.984800 6172 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55950:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 07:59:10.984867 10753 ts_tablet_manager.cc:1875] T ff9dc384dbac47dc9795617c64688943 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:59:10.986023 8924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72aa4dc7664b48b7a73d34feba7d44d3" candidate_uuid: "74077b61d6f54342bf9736b4a9f2eade" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13d05ebdca8c4f489c597ce550dd773d" is_pre_election: true
I20240501 07:59:10.986155 8924 raft_consensus.cc:2454] T 72aa4dc7664b48b7a73d34feba7d44d3 P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74077b61d6f54342bf9736b4a9f2eade in term 0.
I20240501 07:59:10.986449 8729 leader_election.cc:302] T 72aa4dc7664b48b7a73d34feba7d44d3 P 74077b61d6f54342bf9736b4a9f2eade [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13d05ebdca8c4f489c597ce550dd773d, 74077b61d6f54342bf9736b4a9f2eade; no voters:
I20240501 07:59:10.986614 10403 raft_consensus.cc:2781] T 72aa4dc7664b48b7a73d34feba7d44d3 P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:10.986657 10403 raft_consensus.cc:482] T 72aa4dc7664b48b7a73d34feba7d44d3 P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:10.986690 10403 raft_consensus.cc:3037] T 72aa4dc7664b48b7a73d34feba7d44d3 P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:10.987278 10753 ts_tablet_manager.cc:1888] T ff9dc384dbac47dc9795617c64688943 P 3b164cbbc00f4e8eb3254ddd349a3ff9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240501 07:59:10.987341 10753 log.cc:1196] T ff9dc384dbac47dc9795617c64688943 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/ff9dc384dbac47dc9795617c64688943
I20240501 07:59:10.987963 10403 raft_consensus.cc:504] T 72aa4dc7664b48b7a73d34feba7d44d3 P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } }
I20240501 07:59:10.988111 6294 tablet_service.cc:1514] Processing DeleteTablet for tablet 3745acee2b5543b697a3c8f45d2618bb with delete_type TABLET_DATA_TOMBSTONED (TS 8b8b3ad3aa8f4053b2167dd550f2086f not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:45480
I20240501 07:59:10.988296 6160 catalog_manager.cc:4830] TS 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131:34471): tablet ff9dc384dbac47dc9795617c64688943 (table rebalance_test_table_1 [id=d0e1f982440849beb494081bea9cb211]) successfully deleted
I20240501 07:59:10.988343 10403 leader_election.cc:288] T 72aa4dc7664b48b7a73d34feba7d44d3 P 74077b61d6f54342bf9736b4a9f2eade [CANDIDATE]: Term 1 election: Requested vote from peers 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134:40141), a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607)
I20240501 07:59:10.988477 10309 tablet_replica.cc:330] T 3745acee2b5543b697a3c8f45d2618bb P 8b8b3ad3aa8f4053b2167dd550f2086f: stopping tablet replica
I20240501 07:59:10.988601 10309 raft_consensus.cc:2229] T 3745acee2b5543b697a3c8f45d2618bb P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:10.988724 10309 raft_consensus.cc:2258] T 3745acee2b5543b697a3c8f45d2618bb P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:10.989420 10309 ts_tablet_manager.cc:1875] T 3745acee2b5543b697a3c8f45d2618bb P 8b8b3ad3aa8f4053b2167dd550f2086f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:59:10.989987 9189 raft_consensus.cc:1261] T dbbae25711d94ed29223004d019ef377 P 83f20e84af4c40d09a4804b6b26e697f [term 1 FOLLOWER]: Refusing update from remote peer 3b164cbbc00f4e8eb3254ddd349a3ff9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:10.990028 8924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72aa4dc7664b48b7a73d34feba7d44d3" candidate_uuid: "74077b61d6f54342bf9736b4a9f2eade" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13d05ebdca8c4f489c597ce550dd773d"
I20240501 07:59:10.990144 8924 raft_consensus.cc:3037] T 72aa4dc7664b48b7a73d34feba7d44d3 P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:10.991173 8680 consensus_queue.cc:1035] T dbbae25711d94ed29223004d019ef377 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:10.991760 10309 ts_tablet_manager.cc:1888] T 3745acee2b5543b697a3c8f45d2618bb P 8b8b3ad3aa8f4053b2167dd550f2086f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 07:59:10.991828 10309 log.cc:1196] T 3745acee2b5543b697a3c8f45d2618bb P 8b8b3ad3aa8f4053b2167dd550f2086f: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/3745acee2b5543b697a3c8f45d2618bb
I20240501 07:59:10.992727 9940 raft_consensus.cc:482] T 72aa4dc7664b48b7a73d34feba7d44d3 P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:10.992822 9940 raft_consensus.cc:504] T 72aa4dc7664b48b7a73d34feba7d44d3 P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } }
I20240501 07:59:10.993252 8924 raft_consensus.cc:2454] T 72aa4dc7664b48b7a73d34feba7d44d3 P 13d05ebdca8c4f489c597ce550dd773d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74077b61d6f54342bf9736b4a9f2eade in term 1.
I20240501 07:59:10.993299 9940 leader_election.cc:288] T 72aa4dc7664b48b7a73d34feba7d44d3 P a505e7cb133c4392bfda8a4a20874ae9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133:39859), 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134:40141)
I20240501 07:59:10.993592 8729 leader_election.cc:302] T 72aa4dc7664b48b7a73d34feba7d44d3 P 74077b61d6f54342bf9736b4a9f2eade [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13d05ebdca8c4f489c597ce550dd773d, 74077b61d6f54342bf9736b4a9f2eade; no voters:
I20240501 07:59:10.993775 10403 raft_consensus.cc:2781] T 72aa4dc7664b48b7a73d34feba7d44d3 P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:10.993846 10403 raft_consensus.cc:686] T 72aa4dc7664b48b7a73d34feba7d44d3 P 74077b61d6f54342bf9736b4a9f2eade [term 1 LEADER]: Becoming Leader. State: Replica: 74077b61d6f54342bf9736b4a9f2eade, State: Running, Role: LEADER
I20240501 07:59:10.993973 10403 consensus_queue.cc:237] T 72aa4dc7664b48b7a73d34feba7d44d3 P 74077b61d6f54342bf9736b4a9f2eade [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } }
I20240501 07:59:10.995764 8793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72aa4dc7664b48b7a73d34feba7d44d3" candidate_uuid: "a505e7cb133c4392bfda8a4a20874ae9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74077b61d6f54342bf9736b4a9f2eade" is_pre_election: true
I20240501 07:59:10.996555 6166 catalog_manager.cc:5576] TS 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149) does not have the latest schema for tablet dbbae25711d94ed29223004d019ef377 (table rebalancer_dynamic_table [id=2c07df4ff4ae47538ff574e5adb1fd0b]). Expected version 1 got 0
I20240501 07:59:10.999009 6157 catalog_manager.cc:4830] TS 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129:46875): tablet 3745acee2b5543b697a3c8f45d2618bb (table rebalance_test_table_1 [id=d0e1f982440849beb494081bea9cb211]) successfully deleted
I20240501 07:59:10.999083 6160 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 3745acee2b5543b697a3c8f45d2618bb with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 07:59:11.000682 8904 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 07:59:11.001108 9056 raft_consensus.cc:1261] T dbbae25711d94ed29223004d019ef377 P a505e7cb133c4392bfda8a4a20874ae9 [term 1 FOLLOWER]: Refusing update from remote peer 3b164cbbc00f4e8eb3254ddd349a3ff9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:11.001120 8924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72aa4dc7664b48b7a73d34feba7d44d3" candidate_uuid: "a505e7cb133c4392bfda8a4a20874ae9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13d05ebdca8c4f489c597ce550dd773d" is_pre_election: true
I20240501 07:59:11.001327 8924 raft_consensus.cc:2379] T 72aa4dc7664b48b7a73d34feba7d44d3 P 13d05ebdca8c4f489c597ce550dd773d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a505e7cb133c4392bfda8a4a20874ae9 in current term 1: Already voted for candidate 74077b61d6f54342bf9736b4a9f2eade in this term.
I20240501 07:59:11.001688 8991 leader_election.cc:302] T 72aa4dc7664b48b7a73d34feba7d44d3 P a505e7cb133c4392bfda8a4a20874ae9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a505e7cb133c4392bfda8a4a20874ae9; no voters: 13d05ebdca8c4f489c597ce550dd773d, 74077b61d6f54342bf9736b4a9f2eade
I20240501 07:59:11.002117 9940 raft_consensus.cc:3037] T 72aa4dc7664b48b7a73d34feba7d44d3 P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:11.002094 10237 consensus_queue.cc:1035] T dbbae25711d94ed29223004d019ef377 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:11.002928 6168 catalog_manager.cc:5484] T 72aa4dc7664b48b7a73d34feba7d44d3 P 74077b61d6f54342bf9736b4a9f2eade reported cstate change: term changed from 0 to 1, leader changed from <none> to 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133). New cstate: current_term: 1 leader_uuid: "74077b61d6f54342bf9736b4a9f2eade" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } health_report { overall_health: UNKNOWN } } }
I20240501 07:59:11.003294 9940 raft_consensus.cc:2726] T 72aa4dc7664b48b7a73d34feba7d44d3 P a505e7cb133c4392bfda8a4a20874ae9 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 07:59:11.003825 10803 ts_tablet_manager.cc:902] T f68dbc1f65eb4f1b91a62e2de82a7676 P 83f20e84af4c40d09a4804b6b26e697f: Initiating tablet copy from peer 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129:46875)
I20240501 07:59:11.004667 6444 raft_consensus.cc:2932] T 3745acee2b5543b697a3c8f45d2618bb P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false } } }
I20240501 07:59:11.007098 10803 tablet_copy_client.cc:308] T f68dbc1f65eb4f1b91a62e2de82a7676 P 83f20e84af4c40d09a4804b6b26e697f: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:46875
I20240501 07:59:11.008332 6172 catalog_manager.cc:5576] TS a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607) does not have the latest schema for tablet dbbae25711d94ed29223004d019ef377 (table rebalancer_dynamic_table [id=2c07df4ff4ae47538ff574e5adb1fd0b]). Expected version 1 got 0
I20240501 07:59:11.008520 6335 tablet_copy_service.cc:140] P 8b8b3ad3aa8f4053b2167dd550f2086f: Received BeginTabletCopySession request for tablet f68dbc1f65eb4f1b91a62e2de82a7676 from peer 83f20e84af4c40d09a4804b6b26e697f ({username='slave'} at 127.0.99.136:55561)
I20240501 07:59:11.008630 6335 tablet_copy_service.cc:161] P 8b8b3ad3aa8f4053b2167dd550f2086f: Beginning new tablet copy session on tablet f68dbc1f65eb4f1b91a62e2de82a7676 from peer 83f20e84af4c40d09a4804b6b26e697f at {username='slave'} at 127.0.99.136:55561: session id = 83f20e84af4c40d09a4804b6b26e697f-f68dbc1f65eb4f1b91a62e2de82a7676
I20240501 07:59:11.009675 6335 tablet_copy_source_session.cc:205] T f68dbc1f65eb4f1b91a62e2de82a7676 P 8b8b3ad3aa8f4053b2167dd550f2086f: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:11.011276 10803 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f68dbc1f65eb4f1b91a62e2de82a7676. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:11.013415 6172 catalog_manager.cc:5576] TS 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131:34471) does not have the latest schema for tablet dbbae25711d94ed29223004d019ef377 (table rebalancer_dynamic_table [id=2c07df4ff4ae47538ff574e5adb1fd0b]). Expected version 1 got 0
I20240501 07:59:11.016386 6168 catalog_manager.cc:5576] TS 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149) does not have the latest schema for tablet dbbae25711d94ed29223004d019ef377 (table rebalancer_dynamic_table [id=2c07df4ff4ae47538ff574e5adb1fd0b]). Expected version 1 got 0
I20240501 07:59:11.020532 10811 tablet.cc:1722] T dbbae25711d94ed29223004d019ef377 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 07:59:11.021432 9054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72aa4dc7664b48b7a73d34feba7d44d3" candidate_uuid: "74077b61d6f54342bf9736b4a9f2eade" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a505e7cb133c4392bfda8a4a20874ae9"
I20240501 07:59:11.022131 10811 tablet.cc:1620] T dbbae25711d94ed29223004d019ef377 P 3b164cbbc00f4e8eb3254ddd349a3ff9: MemRowSet was empty: no flush needed.
I20240501 07:59:11.022696 9054 raft_consensus.cc:2454] T 72aa4dc7664b48b7a73d34feba7d44d3 P a505e7cb133c4392bfda8a4a20874ae9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74077b61d6f54342bf9736b4a9f2eade in term 1.
I20240501 07:59:11.023626 9055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72aa4dc7664b48b7a73d34feba7d44d3" candidate_uuid: "74077b61d6f54342bf9736b4a9f2eade" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a505e7cb133c4392bfda8a4a20874ae9" is_pre_election: true
I20240501 07:59:11.023743 9055 raft_consensus.cc:2362] T 72aa4dc7664b48b7a73d34feba7d44d3 P a505e7cb133c4392bfda8a4a20874ae9 [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 74077b61d6f54342bf9736b4a9f2eade in term 1. Re-sending same reply.
I20240501 07:59:11.026464 10812 tablet.cc:1722] T dbbae25711d94ed29223004d019ef377 P 83f20e84af4c40d09a4804b6b26e697f: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 07:59:11.026878 10812 tablet.cc:1620] T dbbae25711d94ed29223004d019ef377 P 83f20e84af4c40d09a4804b6b26e697f: MemRowSet was empty: no flush needed.
I20240501 07:59:11.028369 10803 tablet_copy_client.cc:783] T f68dbc1f65eb4f1b91a62e2de82a7676 P 83f20e84af4c40d09a4804b6b26e697f: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:11.028995 8757 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 07:59:11.029974 6425 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 07:59:11.031360 10803 tablet_copy_client.cc:647] T f68dbc1f65eb4f1b91a62e2de82a7676 P 83f20e84af4c40d09a4804b6b26e697f: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:11.033845 6172 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 0bc706e85ce74cef97a4827ea4d4aa65: rebalancer_dynamic_table [id=2c07df4ff4ae47538ff574e5adb1fd0b] alter complete (version 1)
I20240501 07:59:11.035586 10795 ts_tablet_manager.cc:902] T 6292ff0c7e624dd7b24715b83a30fff8 P 1f9cb72be98941b5a8e504cbe34132b9: Initiating tablet copy from peer 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133:39859)
I20240501 07:59:11.036418 10795 tablet_copy_client.cc:308] T 6292ff0c7e624dd7b24715b83a30fff8 P 1f9cb72be98941b5a8e504cbe34132b9: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.133:39859
I20240501 07:59:11.038117 8814 tablet_copy_service.cc:140] P 74077b61d6f54342bf9736b4a9f2eade: Received BeginTabletCopySession request for tablet 6292ff0c7e624dd7b24715b83a30fff8 from peer 1f9cb72be98941b5a8e504cbe34132b9 ({username='slave'} at 127.0.99.132:54045)
I20240501 07:59:11.038249 8814 tablet_copy_service.cc:161] P 74077b61d6f54342bf9736b4a9f2eade: Beginning new tablet copy session on tablet 6292ff0c7e624dd7b24715b83a30fff8 from peer 1f9cb72be98941b5a8e504cbe34132b9 at {username='slave'} at 127.0.99.132:54045: session id = 1f9cb72be98941b5a8e504cbe34132b9-6292ff0c7e624dd7b24715b83a30fff8
I20240501 07:59:11.038748 6687 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 07:59:11.039772 8814 tablet_copy_source_session.cc:205] T 6292ff0c7e624dd7b24715b83a30fff8 P 74077b61d6f54342bf9736b4a9f2eade: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:11.048013 6556 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 07:59:11.053751 10803 tablet_copy_client.cc:523] T f68dbc1f65eb4f1b91a62e2de82a7676 P 83f20e84af4c40d09a4804b6b26e697f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:11.053871 10795 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6292ff0c7e624dd7b24715b83a30fff8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:11.055547 10803 tablet_bootstrap.cc:492] T f68dbc1f65eb4f1b91a62e2de82a7676 P 83f20e84af4c40d09a4804b6b26e697f: Bootstrap starting.
I20240501 07:59:11.056866 10795 tablet_copy_client.cc:783] T 6292ff0c7e624dd7b24715b83a30fff8 P 1f9cb72be98941b5a8e504cbe34132b9: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:11.058364 10810 ts_tablet_manager.cc:902] T 114b6dbafa1c4c44ac42f1d2293e3125 P a505e7cb133c4392bfda8a4a20874ae9: Initiating tablet copy from peer 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131:34471)
I20240501 07:59:11.060122 9167 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 07:59:11.060189 6294 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 07:59:11.063719 6172 catalog_manager.cc:5576] TS a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607) does not have the latest schema for tablet dbbae25711d94ed29223004d019ef377 (table rebalancer_dynamic_table [id=2c07df4ff4ae47538ff574e5adb1fd0b]). Expected version 1 got 0
I20240501 07:59:11.059620 10810 tablet_copy_client.cc:308] T 114b6dbafa1c4c44ac42f1d2293e3125 P a505e7cb133c4392bfda8a4a20874ae9: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:34471
I20240501 07:59:11.068020 10813 ts_tablet_manager.cc:902] T 33b3834db0cd4407ae7c26e044459e87 P a505e7cb133c4392bfda8a4a20874ae9: Initiating tablet copy from peer 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131:34471)
I20240501 07:59:11.078253 6597 tablet_copy_service.cc:140] P 3b164cbbc00f4e8eb3254ddd349a3ff9: Received BeginTabletCopySession request for tablet 114b6dbafa1c4c44ac42f1d2293e3125 from peer a505e7cb133c4392bfda8a4a20874ae9 ({username='slave'} at 127.0.99.135:40821)
I20240501 07:59:11.078378 6597 tablet_copy_service.cc:161] P 3b164cbbc00f4e8eb3254ddd349a3ff9: Beginning new tablet copy session on tablet 114b6dbafa1c4c44ac42f1d2293e3125 from peer a505e7cb133c4392bfda8a4a20874ae9 at {username='slave'} at 127.0.99.135:40821: session id = a505e7cb133c4392bfda8a4a20874ae9-114b6dbafa1c4c44ac42f1d2293e3125
I20240501 07:59:11.078516 10813 tablet_copy_client.cc:308] T 33b3834db0cd4407ae7c26e044459e87 P a505e7cb133c4392bfda8a4a20874ae9: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:34471
I20240501 07:59:11.079474 6597 tablet_copy_source_session.cc:205] T 114b6dbafa1c4c44ac42f1d2293e3125 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:11.080165 6597 tablet_copy_service.cc:140] P 3b164cbbc00f4e8eb3254ddd349a3ff9: Received BeginTabletCopySession request for tablet 33b3834db0cd4407ae7c26e044459e87 from peer a505e7cb133c4392bfda8a4a20874ae9 ({username='slave'} at 127.0.99.135:40821)
I20240501 07:59:11.080260 6597 tablet_copy_service.cc:161] P 3b164cbbc00f4e8eb3254ddd349a3ff9: Beginning new tablet copy session on tablet 33b3834db0cd4407ae7c26e044459e87 from peer a505e7cb133c4392bfda8a4a20874ae9 at {username='slave'} at 127.0.99.135:40821: session id = a505e7cb133c4392bfda8a4a20874ae9-33b3834db0cd4407ae7c26e044459e87
I20240501 07:59:11.081178 6597 tablet_copy_source_session.cc:205] T 33b3834db0cd4407ae7c26e044459e87 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:11.082118 10813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33b3834db0cd4407ae7c26e044459e87. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:11.084297 10824 ts_tablet_manager.cc:902] T b284ed2cb0264ced99a4ffd90983d902 P 13d05ebdca8c4f489c597ce550dd773d: Initiating tablet copy from peer 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132:36773)
I20240501 07:59:11.084642 10813 tablet_copy_client.cc:783] T 33b3834db0cd4407ae7c26e044459e87 P a505e7cb133c4392bfda8a4a20874ae9: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:11.084865 10813 tablet_copy_client.cc:647] T 33b3834db0cd4407ae7c26e044459e87 P a505e7cb133c4392bfda8a4a20874ae9: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:11.086822 10824 tablet_copy_client.cc:308] T b284ed2cb0264ced99a4ffd90983d902 P 13d05ebdca8c4f489c597ce550dd773d: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:36773
I20240501 07:59:11.086839 10352 tablet_bootstrap.cc:492] T 6c3c0d6fe41f4ae196ae29beb9b68125 P a505e7cb133c4392bfda8a4a20874ae9: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:11.087096 10352 tablet_bootstrap.cc:492] T 6c3c0d6fe41f4ae196ae29beb9b68125 P a505e7cb133c4392bfda8a4a20874ae9: Bootstrap complete.
I20240501 07:59:11.087357 10352 ts_tablet_manager.cc:1367] T 6c3c0d6fe41f4ae196ae29beb9b68125 P a505e7cb133c4392bfda8a4a20874ae9: Time spent bootstrapping tablet: real 0.109s user 0.027s sys 0.000s
I20240501 07:59:11.087881 10352 raft_consensus.cc:348] T 6c3c0d6fe41f4ae196ae29beb9b68125 P a505e7cb133c4392bfda8a4a20874ae9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { replace: true } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: true } }
I20240501 07:59:11.089234 10352 raft_consensus.cc:729] T 6c3c0d6fe41f4ae196ae29beb9b68125 P a505e7cb133c4392bfda8a4a20874ae9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a505e7cb133c4392bfda8a4a20874ae9, State: Initialized, Role: LEARNER
I20240501 07:59:11.089707 10352 consensus_queue.cc:260] T 6c3c0d6fe41f4ae196ae29beb9b68125 P a505e7cb133c4392bfda8a4a20874ae9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { replace: true } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: true } }
I20240501 07:59:11.089933 10813 tablet_copy_client.cc:523] T 33b3834db0cd4407ae7c26e044459e87 P a505e7cb133c4392bfda8a4a20874ae9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:11.088243 10829 ts_tablet_manager.cc:902] T e92d6aeeb3df4ea3852c6ebbed0e07c5 P 13d05ebdca8c4f489c597ce550dd773d: Initiating tablet copy from peer 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130:32941)
I20240501 07:59:11.091331 10813 tablet_bootstrap.cc:492] T 33b3834db0cd4407ae7c26e044459e87 P a505e7cb133c4392bfda8a4a20874ae9: Bootstrap starting.
I20240501 07:59:11.091325 10810 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 114b6dbafa1c4c44ac42f1d2293e3125. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:11.092856 10829 tablet_copy_client.cc:308] T e92d6aeeb3df4ea3852c6ebbed0e07c5 P 13d05ebdca8c4f489c597ce550dd773d: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:32941
I20240501 07:59:11.093336 6466 tablet_copy_service.cc:140] P 8315d22c56f7439a9e1fc82f153f018c: Received BeginTabletCopySession request for tablet e92d6aeeb3df4ea3852c6ebbed0e07c5 from peer 13d05ebdca8c4f489c597ce550dd773d ({username='slave'} at 127.0.99.134:41693)
I20240501 07:59:11.093446 6466 tablet_copy_service.cc:161] P 8315d22c56f7439a9e1fc82f153f018c: Beginning new tablet copy session on tablet e92d6aeeb3df4ea3852c6ebbed0e07c5 from peer 13d05ebdca8c4f489c597ce550dd773d at {username='slave'} at 127.0.99.134:41693: session id = 13d05ebdca8c4f489c597ce550dd773d-e92d6aeeb3df4ea3852c6ebbed0e07c5
I20240501 07:59:11.093722 10352 ts_tablet_manager.cc:1398] T 6c3c0d6fe41f4ae196ae29beb9b68125 P a505e7cb133c4392bfda8a4a20874ae9: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20240501 07:59:11.094342 6466 tablet_copy_source_session.cc:205] T e92d6aeeb3df4ea3852c6ebbed0e07c5 P 8315d22c56f7439a9e1fc82f153f018c: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:11.095213 10829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e92d6aeeb3df4ea3852c6ebbed0e07c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:11.096129 10403 raft_consensus.cc:482] T ab1666f1e5204416adb435d9bbe80433 P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:11.096231 10403 raft_consensus.cc:504] T ab1666f1e5204416adb435d9bbe80433 P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } }
I20240501 07:59:11.096625 10403 leader_election.cc:288] T ab1666f1e5204416adb435d9bbe80433 P 74077b61d6f54342bf9736b4a9f2eade [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129:46875), 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149)
I20240501 07:59:11.097388 9189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab1666f1e5204416adb435d9bbe80433" candidate_uuid: "74077b61d6f54342bf9736b4a9f2eade" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83f20e84af4c40d09a4804b6b26e697f" is_pre_election: true
I20240501 07:59:11.097540 9189 raft_consensus.cc:2454] T ab1666f1e5204416adb435d9bbe80433 P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74077b61d6f54342bf9736b4a9f2eade in term 0.
I20240501 07:59:11.097674 6311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab1666f1e5204416adb435d9bbe80433" candidate_uuid: "74077b61d6f54342bf9736b4a9f2eade" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" is_pre_election: true
I20240501 07:59:11.097808 6311 raft_consensus.cc:2454] T ab1666f1e5204416adb435d9bbe80433 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74077b61d6f54342bf9736b4a9f2eade in term 0.
I20240501 07:59:11.098059 9035 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:59:11.098124 8727 leader_election.cc:302] T ab1666f1e5204416adb435d9bbe80433 P 74077b61d6f54342bf9736b4a9f2eade [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74077b61d6f54342bf9736b4a9f2eade, 8b8b3ad3aa8f4053b2167dd550f2086f; no voters:
I20240501 07:59:11.098311 10403 raft_consensus.cc:2781] T ab1666f1e5204416adb435d9bbe80433 P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:11.098364 10403 raft_consensus.cc:482] T ab1666f1e5204416adb435d9bbe80433 P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:11.098409 10403 raft_consensus.cc:3037] T ab1666f1e5204416adb435d9bbe80433 P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:11.098380 6335 tablet_copy_service.cc:342] P 8b8b3ad3aa8f4053b2167dd550f2086f: Request end of tablet copy session a505e7cb133c4392bfda8a4a20874ae9-6c3c0d6fe41f4ae196ae29beb9b68125 received from {username='slave'} at 127.0.99.135:38447
I20240501 07:59:11.098415 10810 tablet_copy_client.cc:783] T 114b6dbafa1c4c44ac42f1d2293e3125 P a505e7cb133c4392bfda8a4a20874ae9: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:11.098462 6335 tablet_copy_service.cc:433] P 8b8b3ad3aa8f4053b2167dd550f2086f: ending tablet copy session a505e7cb133c4392bfda8a4a20874ae9-6c3c0d6fe41f4ae196ae29beb9b68125 on tablet 6c3c0d6fe41f4ae196ae29beb9b68125 with peer a505e7cb133c4392bfda8a4a20874ae9
I20240501 07:59:11.098553 10810 tablet_copy_client.cc:647] T 114b6dbafa1c4c44ac42f1d2293e3125 P a505e7cb133c4392bfda8a4a20874ae9: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:11.099373 10403 raft_consensus.cc:504] T ab1666f1e5204416adb435d9bbe80433 P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } }
I20240501 07:59:11.099722 10403 leader_election.cc:288] T ab1666f1e5204416adb435d9bbe80433 P 74077b61d6f54342bf9736b4a9f2eade [CANDIDATE]: Term 1 election: Requested vote from peers 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129:46875), 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149)
I20240501 07:59:11.100459 6313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab1666f1e5204416adb435d9bbe80433" candidate_uuid: "74077b61d6f54342bf9736b4a9f2eade" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f"
I20240501 07:59:11.100572 6313 raft_consensus.cc:3037] T ab1666f1e5204416adb435d9bbe80433 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:11.101812 6313 raft_consensus.cc:2454] T ab1666f1e5204416adb435d9bbe80433 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74077b61d6f54342bf9736b4a9f2eade in term 1.
I20240501 07:59:11.102152 8727 leader_election.cc:302] T ab1666f1e5204416adb435d9bbe80433 P 74077b61d6f54342bf9736b4a9f2eade [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74077b61d6f54342bf9736b4a9f2eade, 8b8b3ad3aa8f4053b2167dd550f2086f; no voters:
I20240501 07:59:11.102411 10403 raft_consensus.cc:2781] T ab1666f1e5204416adb435d9bbe80433 P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:11.102485 10403 raft_consensus.cc:686] T ab1666f1e5204416adb435d9bbe80433 P 74077b61d6f54342bf9736b4a9f2eade [term 1 LEADER]: Becoming Leader. State: Replica: 74077b61d6f54342bf9736b4a9f2eade, State: Running, Role: LEADER
I20240501 07:59:11.102605 10403 consensus_queue.cc:237] T ab1666f1e5204416adb435d9bbe80433 P 74077b61d6f54342bf9736b4a9f2eade [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } }
I20240501 07:59:11.091886 6728 tablet_copy_service.cc:140] P 1f9cb72be98941b5a8e504cbe34132b9: Received BeginTabletCopySession request for tablet b284ed2cb0264ced99a4ffd90983d902 from peer 13d05ebdca8c4f489c597ce550dd773d ({username='slave'} at 127.0.99.134:44489)
I20240501 07:59:11.104455 6728 tablet_copy_service.cc:161] P 1f9cb72be98941b5a8e504cbe34132b9: Beginning new tablet copy session on tablet b284ed2cb0264ced99a4ffd90983d902 from peer 13d05ebdca8c4f489c597ce550dd773d at {username='slave'} at 127.0.99.134:44489: session id = 13d05ebdca8c4f489c597ce550dd773d-b284ed2cb0264ced99a4ffd90983d902
I20240501 07:59:11.105425 6728 tablet_copy_source_session.cc:205] T b284ed2cb0264ced99a4ffd90983d902 P 1f9cb72be98941b5a8e504cbe34132b9: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:11.105607 9189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab1666f1e5204416adb435d9bbe80433" candidate_uuid: "74077b61d6f54342bf9736b4a9f2eade" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83f20e84af4c40d09a4804b6b26e697f"
I20240501 07:59:11.105726 9189 raft_consensus.cc:3037] T ab1666f1e5204416adb435d9bbe80433 P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:11.106030 10795 tablet_copy_client.cc:647] T 6292ff0c7e624dd7b24715b83a30fff8 P 1f9cb72be98941b5a8e504cbe34132b9: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:11.106643 9189 raft_consensus.cc:2454] T ab1666f1e5204416adb435d9bbe80433 P 83f20e84af4c40d09a4804b6b26e697f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74077b61d6f54342bf9736b4a9f2eade in term 1.
I20240501 07:59:11.106917 10822 tablet.cc:1722] T dbbae25711d94ed29223004d019ef377 P a505e7cb133c4392bfda8a4a20874ae9: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 07:59:11.107236 10822 tablet.cc:1620] T dbbae25711d94ed29223004d019ef377 P a505e7cb133c4392bfda8a4a20874ae9: MemRowSet was empty: no flush needed.
I20240501 07:59:11.107805 6172 catalog_manager.cc:5484] T ab1666f1e5204416adb435d9bbe80433 P 74077b61d6f54342bf9736b4a9f2eade reported cstate change: term changed from 0 to 1, leader changed from <none> to 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133). New cstate: current_term: 1 leader_uuid: "74077b61d6f54342bf9736b4a9f2eade" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } health_report { overall_health: HEALTHY } } }
I20240501 07:59:11.114064 10824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b284ed2cb0264ced99a4ffd90983d902. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:11.114745 10810 tablet_copy_client.cc:523] T 114b6dbafa1c4c44ac42f1d2293e3125 P a505e7cb133c4392bfda8a4a20874ae9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:11.116147 10810 tablet_bootstrap.cc:492] T 114b6dbafa1c4c44ac42f1d2293e3125 P a505e7cb133c4392bfda8a4a20874ae9: Bootstrap starting.
I20240501 07:59:11.117447 10824 tablet_copy_client.cc:783] T b284ed2cb0264ced99a4ffd90983d902 P 13d05ebdca8c4f489c597ce550dd773d: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:11.117652 10824 tablet_copy_client.cc:647] T b284ed2cb0264ced99a4ffd90983d902 P 13d05ebdca8c4f489c597ce550dd773d: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:11.118846 10829 tablet_copy_client.cc:783] T e92d6aeeb3df4ea3852c6ebbed0e07c5 P 13d05ebdca8c4f489c597ce550dd773d: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:11.119029 10829 tablet_copy_client.cc:647] T e92d6aeeb3df4ea3852c6ebbed0e07c5 P 13d05ebdca8c4f489c597ce550dd773d: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:11.120040 10824 tablet_copy_client.cc:523] T b284ed2cb0264ced99a4ffd90983d902 P 13d05ebdca8c4f489c597ce550dd773d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:11.120258 10795 tablet_copy_client.cc:523] T 6292ff0c7e624dd7b24715b83a30fff8 P 1f9cb72be98941b5a8e504cbe34132b9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:11.121698 10824 tablet_bootstrap.cc:492] T b284ed2cb0264ced99a4ffd90983d902 P 13d05ebdca8c4f489c597ce550dd773d: Bootstrap starting.
I20240501 07:59:11.122125 10795 tablet_bootstrap.cc:492] T 6292ff0c7e624dd7b24715b83a30fff8 P 1f9cb72be98941b5a8e504cbe34132b9: Bootstrap starting.
I20240501 07:59:11.123155 9618 raft_consensus.cc:1050] T 72feac2a993141168ac9399d9361ef17 P 1f9cb72be98941b5a8e504cbe34132b9: attempting to promote NON_VOTER 13d05ebdca8c4f489c597ce550dd773d to VOTER
I20240501 07:59:11.123579 9618 consensus_queue.cc:237] T 72feac2a993141168ac9399d9361ef17 P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false } }
I20240501 07:59:11.125319 6444 raft_consensus.cc:1261] T 72feac2a993141168ac9399d9361ef17 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Refusing update from remote peer 1f9cb72be98941b5a8e504cbe34132b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:59:11.125419 6577 raft_consensus.cc:1261] T 72feac2a993141168ac9399d9361ef17 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Refusing update from remote peer 1f9cb72be98941b5a8e504cbe34132b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:59:11.125720 10288 consensus_queue.cc:1035] T 72feac2a993141168ac9399d9361ef17 P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:59:11.125766 8924 raft_consensus.cc:1261] T 72feac2a993141168ac9399d9361ef17 P 13d05ebdca8c4f489c597ce550dd773d [term 1 LEARNER]: Refusing update from remote peer 1f9cb72be98941b5a8e504cbe34132b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:59:11.126636 10324 consensus_queue.cc:1035] T 72feac2a993141168ac9399d9361ef17 P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:59:11.126996 9990 consensus_queue.cc:1035] T 72feac2a993141168ac9399d9361ef17 P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:59:11.130225 10829 tablet_copy_client.cc:523] T e92d6aeeb3df4ea3852c6ebbed0e07c5 P 13d05ebdca8c4f489c597ce550dd773d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:11.132552 10829 tablet_bootstrap.cc:492] T e92d6aeeb3df4ea3852c6ebbed0e07c5 P 13d05ebdca8c4f489c597ce550dd773d: Bootstrap starting.
I20240501 07:59:11.133816 10795 tablet_bootstrap.cc:492] T 6292ff0c7e624dd7b24715b83a30fff8 P 1f9cb72be98941b5a8e504cbe34132b9: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:11.134101 10795 tablet_bootstrap.cc:492] T 6292ff0c7e624dd7b24715b83a30fff8 P 1f9cb72be98941b5a8e504cbe34132b9: Bootstrap complete.
I20240501 07:59:11.134267 10795 ts_tablet_manager.cc:1367] T 6292ff0c7e624dd7b24715b83a30fff8 P 1f9cb72be98941b5a8e504cbe34132b9: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.002s
I20240501 07:59:11.134699 10795 raft_consensus.cc:348] T 6292ff0c7e624dd7b24715b83a30fff8 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { replace: true } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { promote: true } }
I20240501 07:59:11.134814 10795 raft_consensus.cc:729] T 6292ff0c7e624dd7b24715b83a30fff8 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1f9cb72be98941b5a8e504cbe34132b9, State: Initialized, Role: LEARNER
I20240501 07:59:11.134907 10795 consensus_queue.cc:260] T 6292ff0c7e624dd7b24715b83a30fff8 P 1f9cb72be98941b5a8e504cbe34132b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { replace: true } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { promote: true } }
I20240501 07:59:11.135177 10795 ts_tablet_manager.cc:1398] T 6292ff0c7e624dd7b24715b83a30fff8 P 1f9cb72be98941b5a8e504cbe34132b9: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 07:59:11.135756 8814 tablet_copy_service.cc:342] P 74077b61d6f54342bf9736b4a9f2eade: Request end of tablet copy session 1f9cb72be98941b5a8e504cbe34132b9-6292ff0c7e624dd7b24715b83a30fff8 received from {username='slave'} at 127.0.99.132:54045
I20240501 07:59:11.135849 8814 tablet_copy_service.cc:433] P 74077b61d6f54342bf9736b4a9f2eade: ending tablet copy session 1f9cb72be98941b5a8e504cbe34132b9-6292ff0c7e624dd7b24715b83a30fff8 on tablet 6292ff0c7e624dd7b24715b83a30fff8 with peer 1f9cb72be98941b5a8e504cbe34132b9
I20240501 07:59:11.140465 10813 tablet_bootstrap.cc:492] T 33b3834db0cd4407ae7c26e044459e87 P a505e7cb133c4392bfda8a4a20874ae9: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:11.140868 10813 tablet_bootstrap.cc:492] T 33b3834db0cd4407ae7c26e044459e87 P a505e7cb133c4392bfda8a4a20874ae9: Bootstrap complete.
I20240501 07:59:11.141140 10813 ts_tablet_manager.cc:1367] T 33b3834db0cd4407ae7c26e044459e87 P a505e7cb133c4392bfda8a4a20874ae9: Time spent bootstrapping tablet: real 0.050s user 0.030s sys 0.000s
I20240501 07:59:11.141649 10813 raft_consensus.cc:348] T 33b3834db0cd4407ae7c26e044459e87 P a505e7cb133c4392bfda8a4a20874ae9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: true } }
I20240501 07:59:11.141849 10813 raft_consensus.cc:729] T 33b3834db0cd4407ae7c26e044459e87 P a505e7cb133c4392bfda8a4a20874ae9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a505e7cb133c4392bfda8a4a20874ae9, State: Initialized, Role: LEARNER
I20240501 07:59:11.141894 6172 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55950:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 07:59:11.141988 10813 consensus_queue.cc:260] T 33b3834db0cd4407ae7c26e044459e87 P a505e7cb133c4392bfda8a4a20874ae9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: true } }
I20240501 07:59:11.147579 10813 ts_tablet_manager.cc:1398] T 33b3834db0cd4407ae7c26e044459e87 P a505e7cb133c4392bfda8a4a20874ae9: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20240501 07:59:11.149667 10811 tablet.cc:1722] T dbbae25711d94ed29223004d019ef377 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 07:59:11.150012 10811 tablet.cc:1620] T dbbae25711d94ed29223004d019ef377 P 3b164cbbc00f4e8eb3254ddd349a3ff9: MemRowSet was empty: no flush needed.
I20240501 07:59:11.151758 6597 tablet_copy_service.cc:342] P 3b164cbbc00f4e8eb3254ddd349a3ff9: Request end of tablet copy session a505e7cb133c4392bfda8a4a20874ae9-33b3834db0cd4407ae7c26e044459e87 received from {username='slave'} at 127.0.99.135:40821
I20240501 07:59:11.151840 6597 tablet_copy_service.cc:433] P 3b164cbbc00f4e8eb3254ddd349a3ff9: ending tablet copy session a505e7cb133c4392bfda8a4a20874ae9-33b3834db0cd4407ae7c26e044459e87 on tablet 33b3834db0cd4407ae7c26e044459e87 with peer a505e7cb133c4392bfda8a4a20874ae9
I20240501 07:59:11.154924 10830 ts_tablet_manager.cc:902] T 2e540f389b2a4aef93a020f549d11b67 P 83f20e84af4c40d09a4804b6b26e697f: Initiating tablet copy from peer 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129:46875)
I20240501 07:59:11.155474 10830 tablet_copy_client.cc:308] T 2e540f389b2a4aef93a020f549d11b67 P 83f20e84af4c40d09a4804b6b26e697f: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:46875
I20240501 07:59:11.155967 6335 tablet_copy_service.cc:140] P 8b8b3ad3aa8f4053b2167dd550f2086f: Received BeginTabletCopySession request for tablet 2e540f389b2a4aef93a020f549d11b67 from peer 83f20e84af4c40d09a4804b6b26e697f ({username='slave'} at 127.0.99.136:55561)
I20240501 07:59:11.156070 6335 tablet_copy_service.cc:161] P 8b8b3ad3aa8f4053b2167dd550f2086f: Beginning new tablet copy session on tablet 2e540f389b2a4aef93a020f549d11b67 from peer 83f20e84af4c40d09a4804b6b26e697f at {username='slave'} at 127.0.99.136:55561: session id = 83f20e84af4c40d09a4804b6b26e697f-2e540f389b2a4aef93a020f549d11b67
I20240501 07:59:11.156066 10822 tablet.cc:1722] T dbbae25711d94ed29223004d019ef377 P a505e7cb133c4392bfda8a4a20874ae9: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 07:59:11.156368 10822 tablet.cc:1620] T dbbae25711d94ed29223004d019ef377 P a505e7cb133c4392bfda8a4a20874ae9: MemRowSet was empty: no flush needed.
I20240501 07:59:11.156538 10812 tablet.cc:1722] T dbbae25711d94ed29223004d019ef377 P 83f20e84af4c40d09a4804b6b26e697f: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 07:59:11.156816 10812 tablet.cc:1620] T dbbae25711d94ed29223004d019ef377 P 83f20e84af4c40d09a4804b6b26e697f: MemRowSet was empty: no flush needed.
I20240501 07:59:11.157040 6335 tablet_copy_source_session.cc:205] T 2e540f389b2a4aef93a020f549d11b67 P 8b8b3ad3aa8f4053b2167dd550f2086f: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:11.157999 10830 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e540f389b2a4aef93a020f549d11b67. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:11.161063 10830 tablet_copy_client.cc:783] T 2e540f389b2a4aef93a020f549d11b67 P 83f20e84af4c40d09a4804b6b26e697f: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:11.161139 6168 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 0bc706e85ce74cef97a4827ea4d4aa65: rebalancer_dynamic_table [id=2c07df4ff4ae47538ff574e5adb1fd0b] alter complete (version 2)
I20240501 07:59:11.167439 10840 ts_tablet_manager.cc:902] T fb6b46072a8d46e18b53b9f8de702859 P 8b8b3ad3aa8f4053b2167dd550f2086f: Initiating tablet copy from peer 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134:40141)
I20240501 07:59:11.168416 10840 tablet_copy_client.cc:308] T fb6b46072a8d46e18b53b9f8de702859 P 8b8b3ad3aa8f4053b2167dd550f2086f: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.134:40141
I20240501 07:59:11.169286 8945 tablet_copy_service.cc:140] P 13d05ebdca8c4f489c597ce550dd773d: Received BeginTabletCopySession request for tablet fb6b46072a8d46e18b53b9f8de702859 from peer 8b8b3ad3aa8f4053b2167dd550f2086f ({username='slave'} at 127.0.99.129:40355)
I20240501 07:59:11.169417 8945 tablet_copy_service.cc:161] P 13d05ebdca8c4f489c597ce550dd773d: Beginning new tablet copy session on tablet fb6b46072a8d46e18b53b9f8de702859 from peer 8b8b3ad3aa8f4053b2167dd550f2086f at {username='slave'} at 127.0.99.129:40355: session id = 8b8b3ad3aa8f4053b2167dd550f2086f-fb6b46072a8d46e18b53b9f8de702859
I20240501 07:59:11.171243 8945 tablet_copy_source_session.cc:205] T fb6b46072a8d46e18b53b9f8de702859 P 13d05ebdca8c4f489c597ce550dd773d: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:11.172890 9990 raft_consensus.cc:2932] T 72feac2a993141168ac9399d9361ef17 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false } } }
I20240501 07:59:11.175051 6577 raft_consensus.cc:2932] T 72feac2a993141168ac9399d9361ef17 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false } } }
I20240501 07:59:11.175856 6444 raft_consensus.cc:2932] T 72feac2a993141168ac9399d9361ef17 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false } } }
I20240501 07:59:11.176472 6172 catalog_manager.cc:5484] T 72feac2a993141168ac9399d9361ef17 P 1f9cb72be98941b5a8e504cbe34132b9 reported cstate change: config changed from index 4 to 5, 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1f9cb72be98941b5a8e504cbe34132b9" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 07:59:11.179735 10840 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb6b46072a8d46e18b53b9f8de702859. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:11.181913 10830 tablet_copy_client.cc:647] T 2e540f389b2a4aef93a020f549d11b67 P 83f20e84af4c40d09a4804b6b26e697f: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:11.184010 8924 raft_consensus.cc:2932] T 72feac2a993141168ac9399d9361ef17 P 13d05ebdca8c4f489c597ce550dd773d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false } } }
I20240501 07:59:11.186100 10830 tablet_copy_client.cc:523] T 2e540f389b2a4aef93a020f549d11b67 P 83f20e84af4c40d09a4804b6b26e697f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:11.187778 6708 consensus_queue.cc:237] T 72feac2a993141168ac9399d9361ef17 P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false } }
I20240501 07:59:11.188184 10830 tablet_bootstrap.cc:492] T 2e540f389b2a4aef93a020f549d11b67 P 83f20e84af4c40d09a4804b6b26e697f: Bootstrap starting.
I20240501 07:59:11.189188 6577 raft_consensus.cc:1261] T 72feac2a993141168ac9399d9361ef17 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Refusing update from remote peer 1f9cb72be98941b5a8e504cbe34132b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 07:59:11.189591 10324 consensus_queue.cc:1035] T 72feac2a993141168ac9399d9361ef17 P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 07:59:11.191542 10324 raft_consensus.cc:2932] T 72feac2a993141168ac9399d9361ef17 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false } } }
I20240501 07:59:11.193713 6159 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 72feac2a993141168ac9399d9361ef17 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 07:59:11.194020 8924 raft_consensus.cc:1261] T 72feac2a993141168ac9399d9361ef17 P 13d05ebdca8c4f489c597ce550dd773d [term 1 FOLLOWER]: Refusing update from remote peer 1f9cb72be98941b5a8e504cbe34132b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 07:59:11.194056 6577 raft_consensus.cc:2932] T 72feac2a993141168ac9399d9361ef17 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false } } }
I20240501 07:59:11.194671 9618 consensus_queue.cc:1035] T 72feac2a993141168ac9399d9361ef17 P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 07:59:11.195807 8924 raft_consensus.cc:2932] T 72feac2a993141168ac9399d9361ef17 P 13d05ebdca8c4f489c597ce550dd773d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false } } }
I20240501 07:59:11.197490 6168 catalog_manager.cc:5484] T 72feac2a993141168ac9399d9361ef17 P 3b164cbbc00f4e8eb3254ddd349a3ff9 reported cstate change: config changed from index 5 to 6, VOTER 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "1f9cb72be98941b5a8e504cbe34132b9" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false } } }
I20240501 07:59:11.200611 6425 tablet_service.cc:1514] Processing DeleteTablet for tablet 72feac2a993141168ac9399d9361ef17 with delete_type TABLET_DATA_TOMBSTONED (TS 8315d22c56f7439a9e1fc82f153f018c not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:45034
I20240501 07:59:11.200860 10751 tablet_replica.cc:330] T 72feac2a993141168ac9399d9361ef17 P 8315d22c56f7439a9e1fc82f153f018c: stopping tablet replica
I20240501 07:59:11.201043 10751 raft_consensus.cc:2229] T 72feac2a993141168ac9399d9361ef17 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:11.201234 10751 raft_consensus.cc:2258] T 72feac2a993141168ac9399d9361ef17 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:11.172394 10803 tablet_bootstrap.cc:492] T f68dbc1f65eb4f1b91a62e2de82a7676 P 83f20e84af4c40d09a4804b6b26e697f: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:11.201972 10803 tablet_bootstrap.cc:492] T f68dbc1f65eb4f1b91a62e2de82a7676 P 83f20e84af4c40d09a4804b6b26e697f: Bootstrap complete.
I20240501 07:59:11.202237 10803 ts_tablet_manager.cc:1367] T f68dbc1f65eb4f1b91a62e2de82a7676 P 83f20e84af4c40d09a4804b6b26e697f: Time spent bootstrapping tablet: real 0.147s user 0.029s sys 0.000s
I20240501 07:59:11.202283 10751 ts_tablet_manager.cc:1875] T 72feac2a993141168ac9399d9361ef17 P 8315d22c56f7439a9e1fc82f153f018c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:59:11.202811 10803 raft_consensus.cc:348] T f68dbc1f65eb4f1b91a62e2de82a7676 P 83f20e84af4c40d09a4804b6b26e697f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: true } }
I20240501 07:59:11.204648 10751 ts_tablet_manager.cc:1888] T 72feac2a993141168ac9399d9361ef17 P 8315d22c56f7439a9e1fc82f153f018c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 07:59:11.204735 10751 log.cc:1196] T 72feac2a993141168ac9399d9361ef17 P 8315d22c56f7439a9e1fc82f153f018c: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/72feac2a993141168ac9399d9361ef17
I20240501 07:59:11.204849 10803 raft_consensus.cc:729] T f68dbc1f65eb4f1b91a62e2de82a7676 P 83f20e84af4c40d09a4804b6b26e697f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 83f20e84af4c40d09a4804b6b26e697f, State: Initialized, Role: LEARNER
I20240501 07:59:11.205119 10803 consensus_queue.cc:260] T f68dbc1f65eb4f1b91a62e2de82a7676 P 83f20e84af4c40d09a4804b6b26e697f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: true } }
I20240501 07:59:11.206975 10803 ts_tablet_manager.cc:1398] T f68dbc1f65eb4f1b91a62e2de82a7676 P 83f20e84af4c40d09a4804b6b26e697f: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240501 07:59:11.207791 6335 tablet_copy_service.cc:342] P 8b8b3ad3aa8f4053b2167dd550f2086f: Request end of tablet copy session 83f20e84af4c40d09a4804b6b26e697f-f68dbc1f65eb4f1b91a62e2de82a7676 received from {username='slave'} at 127.0.99.136:55561
I20240501 07:59:11.207886 6335 tablet_copy_service.cc:433] P 8b8b3ad3aa8f4053b2167dd550f2086f: ending tablet copy session 83f20e84af4c40d09a4804b6b26e697f-f68dbc1f65eb4f1b91a62e2de82a7676 on tablet f68dbc1f65eb4f1b91a62e2de82a7676 with peer 83f20e84af4c40d09a4804b6b26e697f
I20240501 07:59:11.210024 6159 catalog_manager.cc:4830] TS 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130:32941): tablet 72feac2a993141168ac9399d9361ef17 (table rebalance_test_table_9 [id=c4f46595186f4427aa00d6752aa5dc9a]) successfully deleted
I20240501 07:59:11.212538 8793 raft_consensus.cc:1203] T 2398430a23094711a13a86cc88e27a78 P 74077b61d6f54342bf9736b4a9f2eade [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 07:59:11.221778 10840 tablet_copy_client.cc:783] T fb6b46072a8d46e18b53b9f8de702859 P 8b8b3ad3aa8f4053b2167dd550f2086f: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:11.222576 10840 tablet_copy_client.cc:647] T fb6b46072a8d46e18b53b9f8de702859 P 8b8b3ad3aa8f4053b2167dd550f2086f: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:11.224637 10840 tablet_copy_client.cc:523] T fb6b46072a8d46e18b53b9f8de702859 P 8b8b3ad3aa8f4053b2167dd550f2086f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:11.228261 10824 tablet_bootstrap.cc:492] T b284ed2cb0264ced99a4ffd90983d902 P 13d05ebdca8c4f489c597ce550dd773d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:11.228652 10824 tablet_bootstrap.cc:492] T b284ed2cb0264ced99a4ffd90983d902 P 13d05ebdca8c4f489c597ce550dd773d: Bootstrap complete.
I20240501 07:59:11.228834 10824 ts_tablet_manager.cc:1367] T b284ed2cb0264ced99a4ffd90983d902 P 13d05ebdca8c4f489c597ce550dd773d: Time spent bootstrapping tablet: real 0.107s user 0.027s sys 0.000s
I20240501 07:59:11.229362 10824 raft_consensus.cc:348] T b284ed2cb0264ced99a4ffd90983d902 P 13d05ebdca8c4f489c597ce550dd773d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { replace: true } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: true } }
I20240501 07:59:11.229508 10824 raft_consensus.cc:729] T b284ed2cb0264ced99a4ffd90983d902 P 13d05ebdca8c4f489c597ce550dd773d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 13d05ebdca8c4f489c597ce550dd773d, State: Initialized, Role: LEARNER
I20240501 07:59:11.229619 10824 consensus_queue.cc:260] T b284ed2cb0264ced99a4ffd90983d902 P 13d05ebdca8c4f489c597ce550dd773d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { replace: true } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: true } }
I20240501 07:59:11.229900 10824 ts_tablet_manager.cc:1398] T b284ed2cb0264ced99a4ffd90983d902 P 13d05ebdca8c4f489c597ce550dd773d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:59:11.230603 10840 tablet_bootstrap.cc:492] T fb6b46072a8d46e18b53b9f8de702859 P 8b8b3ad3aa8f4053b2167dd550f2086f: Bootstrap starting.
I20240501 07:59:11.230727 6728 tablet_copy_service.cc:342] P 1f9cb72be98941b5a8e504cbe34132b9: Request end of tablet copy session 13d05ebdca8c4f489c597ce550dd773d-b284ed2cb0264ced99a4ffd90983d902 received from {username='slave'} at 127.0.99.134:44489
I20240501 07:59:11.230814 6728 tablet_copy_service.cc:433] P 1f9cb72be98941b5a8e504cbe34132b9: ending tablet copy session 13d05ebdca8c4f489c597ce550dd773d-b284ed2cb0264ced99a4ffd90983d902 on tablet b284ed2cb0264ced99a4ffd90983d902 with peer 13d05ebdca8c4f489c597ce550dd773d
I20240501 07:59:11.232403 10810 tablet_bootstrap.cc:492] T 114b6dbafa1c4c44ac42f1d2293e3125 P a505e7cb133c4392bfda8a4a20874ae9: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:11.232784 10810 tablet_bootstrap.cc:492] T 114b6dbafa1c4c44ac42f1d2293e3125 P a505e7cb133c4392bfda8a4a20874ae9: Bootstrap complete.
I20240501 07:59:11.233057 10810 ts_tablet_manager.cc:1367] T 114b6dbafa1c4c44ac42f1d2293e3125 P a505e7cb133c4392bfda8a4a20874ae9: Time spent bootstrapping tablet: real 0.117s user 0.021s sys 0.005s
I20240501 07:59:11.233556 10810 raft_consensus.cc:348] T 114b6dbafa1c4c44ac42f1d2293e3125 P a505e7cb133c4392bfda8a4a20874ae9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: true } }
I20240501 07:59:11.233850 10810 raft_consensus.cc:729] T 114b6dbafa1c4c44ac42f1d2293e3125 P a505e7cb133c4392bfda8a4a20874ae9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a505e7cb133c4392bfda8a4a20874ae9, State: Initialized, Role: LEARNER
I20240501 07:59:11.234020 10810 consensus_queue.cc:260] T 114b6dbafa1c4c44ac42f1d2293e3125 P a505e7cb133c4392bfda8a4a20874ae9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: true } }
I20240501 07:59:11.236893 10810 ts_tablet_manager.cc:1398] T 114b6dbafa1c4c44ac42f1d2293e3125 P a505e7cb133c4392bfda8a4a20874ae9: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20240501 07:59:11.237803 6597 tablet_copy_service.cc:342] P 3b164cbbc00f4e8eb3254ddd349a3ff9: Request end of tablet copy session a505e7cb133c4392bfda8a4a20874ae9-114b6dbafa1c4c44ac42f1d2293e3125 received from {username='slave'} at 127.0.99.135:40821
I20240501 07:59:11.237884 6597 tablet_copy_service.cc:433] P 3b164cbbc00f4e8eb3254ddd349a3ff9: ending tablet copy session a505e7cb133c4392bfda8a4a20874ae9-114b6dbafa1c4c44ac42f1d2293e3125 on tablet 114b6dbafa1c4c44ac42f1d2293e3125 with peer a505e7cb133c4392bfda8a4a20874ae9
I20240501 07:59:11.242664 10840 tablet_bootstrap.cc:492] T fb6b46072a8d46e18b53b9f8de702859 P 8b8b3ad3aa8f4053b2167dd550f2086f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:11.242996 10840 tablet_bootstrap.cc:492] T fb6b46072a8d46e18b53b9f8de702859 P 8b8b3ad3aa8f4053b2167dd550f2086f: Bootstrap complete.
I20240501 07:59:11.243198 10840 ts_tablet_manager.cc:1367] T fb6b46072a8d46e18b53b9f8de702859 P 8b8b3ad3aa8f4053b2167dd550f2086f: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.001s
I20240501 07:59:11.243937 10840 raft_consensus.cc:348] T fb6b46072a8d46e18b53b9f8de702859 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { replace: true } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { promote: true } }
I20240501 07:59:11.244073 10840 raft_consensus.cc:729] T fb6b46072a8d46e18b53b9f8de702859 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8b8b3ad3aa8f4053b2167dd550f2086f, State: Initialized, Role: LEARNER
I20240501 07:59:11.244199 10840 consensus_queue.cc:260] T fb6b46072a8d46e18b53b9f8de702859 P 8b8b3ad3aa8f4053b2167dd550f2086f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { replace: true } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { promote: true } }
I20240501 07:59:11.245398 10840 ts_tablet_manager.cc:1398] T fb6b46072a8d46e18b53b9f8de702859 P 8b8b3ad3aa8f4053b2167dd550f2086f: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20240501 07:59:11.246011 8945 tablet_copy_service.cc:342] P 13d05ebdca8c4f489c597ce550dd773d: Request end of tablet copy session 8b8b3ad3aa8f4053b2167dd550f2086f-fb6b46072a8d46e18b53b9f8de702859 received from {username='slave'} at 127.0.99.129:40355
I20240501 07:59:11.246112 8945 tablet_copy_service.cc:433] P 13d05ebdca8c4f489c597ce550dd773d: ending tablet copy session 8b8b3ad3aa8f4053b2167dd550f2086f-fb6b46072a8d46e18b53b9f8de702859 on tablet fb6b46072a8d46e18b53b9f8de702859 with peer 8b8b3ad3aa8f4053b2167dd550f2086f
I20240501 07:59:11.247548 10829 tablet_bootstrap.cc:492] T e92d6aeeb3df4ea3852c6ebbed0e07c5 P 13d05ebdca8c4f489c597ce550dd773d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:11.247864 10829 tablet_bootstrap.cc:492] T e92d6aeeb3df4ea3852c6ebbed0e07c5 P 13d05ebdca8c4f489c597ce550dd773d: Bootstrap complete.
I20240501 07:59:11.248075 10829 ts_tablet_manager.cc:1367] T e92d6aeeb3df4ea3852c6ebbed0e07c5 P 13d05ebdca8c4f489c597ce550dd773d: Time spent bootstrapping tablet: real 0.116s user 0.024s sys 0.004s
I20240501 07:59:11.248575 10829 raft_consensus.cc:348] T e92d6aeeb3df4ea3852c6ebbed0e07c5 P 13d05ebdca8c4f489c597ce550dd773d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: true } }
I20240501 07:59:11.248716 10829 raft_consensus.cc:729] T e92d6aeeb3df4ea3852c6ebbed0e07c5 P 13d05ebdca8c4f489c597ce550dd773d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 13d05ebdca8c4f489c597ce550dd773d, State: Initialized, Role: LEARNER
I20240501 07:59:11.248840 10829 consensus_queue.cc:260] T e92d6aeeb3df4ea3852c6ebbed0e07c5 P 13d05ebdca8c4f489c597ce550dd773d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: true } }
I20240501 07:59:11.250063 10829 ts_tablet_manager.cc:1398] T e92d6aeeb3df4ea3852c6ebbed0e07c5 P 13d05ebdca8c4f489c597ce550dd773d: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 07:59:11.250599 6466 tablet_copy_service.cc:342] P 8315d22c56f7439a9e1fc82f153f018c: Request end of tablet copy session 13d05ebdca8c4f489c597ce550dd773d-e92d6aeeb3df4ea3852c6ebbed0e07c5 received from {username='slave'} at 127.0.99.134:41693
I20240501 07:59:11.250689 6466 tablet_copy_service.cc:433] P 8315d22c56f7439a9e1fc82f153f018c: ending tablet copy session 13d05ebdca8c4f489c597ce550dd773d-e92d6aeeb3df4ea3852c6ebbed0e07c5 on tablet e92d6aeeb3df4ea3852c6ebbed0e07c5 with peer 13d05ebdca8c4f489c597ce550dd773d
I20240501 07:59:11.252440 6313 consensus_queue.cc:237] T aa1b8716228b4fd9af95870b7a613fb2 P 8b8b3ad3aa8f4053b2167dd550f2086f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { replace: true } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: true } }
I20240501 07:59:11.254660 6444 raft_consensus.cc:1261] T aa1b8716228b4fd9af95870b7a613fb2 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Refusing update from remote peer 8b8b3ad3aa8f4053b2167dd550f2086f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:59:11.255041 6708 raft_consensus.cc:1261] T aa1b8716228b4fd9af95870b7a613fb2 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Refusing update from remote peer 8b8b3ad3aa8f4053b2167dd550f2086f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:59:11.255630 9964 consensus_queue.cc:1035] T aa1b8716228b4fd9af95870b7a613fb2 P 8b8b3ad3aa8f4053b2167dd550f2086f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 07:59:11.256150 6249 consensus_peers.cc:485] T aa1b8716228b4fd9af95870b7a613fb2 P 8b8b3ad3aa8f4053b2167dd550f2086f -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: aa1b8716228b4fd9af95870b7a613fb2. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:11.256572 9964 consensus_queue.cc:1035] T aa1b8716228b4fd9af95870b7a613fb2 P 8b8b3ad3aa8f4053b2167dd550f2086f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:11.258630 9964 raft_consensus.cc:2932] T aa1b8716228b4fd9af95870b7a613fb2 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { replace: true } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: true } } }
I20240501 07:59:11.260342 6444 raft_consensus.cc:2932] T aa1b8716228b4fd9af95870b7a613fb2 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { replace: true } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: true } } }
I20240501 07:59:11.261651 6164 catalog_manager.cc:5484] T aa1b8716228b4fd9af95870b7a613fb2 P 8b8b3ad3aa8f4053b2167dd550f2086f reported cstate change: config changed from index -1 to 4, NON_VOTER 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 07:59:11.265947 6164 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55950:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 07:59:11.267588 10830 tablet_bootstrap.cc:492] T 2e540f389b2a4aef93a020f549d11b67 P 83f20e84af4c40d09a4804b6b26e697f: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:11.267993 10830 tablet_bootstrap.cc:492] T 2e540f389b2a4aef93a020f549d11b67 P 83f20e84af4c40d09a4804b6b26e697f: Bootstrap complete.
I20240501 07:59:11.268220 10830 ts_tablet_manager.cc:1367] T 2e540f389b2a4aef93a020f549d11b67 P 83f20e84af4c40d09a4804b6b26e697f: Time spent bootstrapping tablet: real 0.080s user 0.027s sys 0.000s
I20240501 07:59:11.268045 6708 raft_consensus.cc:2932] T aa1b8716228b4fd9af95870b7a613fb2 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { replace: true } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: true } } }
I20240501 07:59:11.268746 10830 raft_consensus.cc:348] T 2e540f389b2a4aef93a020f549d11b67 P 83f20e84af4c40d09a4804b6b26e697f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: true } }
I20240501 07:59:11.268875 10830 raft_consensus.cc:729] T 2e540f389b2a4aef93a020f549d11b67 P 83f20e84af4c40d09a4804b6b26e697f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 83f20e84af4c40d09a4804b6b26e697f, State: Initialized, Role: LEARNER
I20240501 07:59:11.269049 10830 consensus_queue.cc:260] T 2e540f389b2a4aef93a020f549d11b67 P 83f20e84af4c40d09a4804b6b26e697f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: true } }
I20240501 07:59:11.269948 10830 ts_tablet_manager.cc:1398] T 2e540f389b2a4aef93a020f549d11b67 P 83f20e84af4c40d09a4804b6b26e697f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:59:11.271363 6335 tablet_copy_service.cc:342] P 8b8b3ad3aa8f4053b2167dd550f2086f: Request end of tablet copy session 83f20e84af4c40d09a4804b6b26e697f-2e540f389b2a4aef93a020f549d11b67 received from {username='slave'} at 127.0.99.136:55561
I20240501 07:59:11.271451 6335 tablet_copy_service.cc:433] P 8b8b3ad3aa8f4053b2167dd550f2086f: ending tablet copy session 83f20e84af4c40d09a4804b6b26e697f-2e540f389b2a4aef93a020f549d11b67 on tablet 2e540f389b2a4aef93a020f549d11b67 with peer 83f20e84af4c40d09a4804b6b26e697f
I20240501 07:59:11.273443 10811 tablet.cc:1722] T dbbae25711d94ed29223004d019ef377 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 07:59:11.273850 10822 tablet.cc:1722] T dbbae25711d94ed29223004d019ef377 P a505e7cb133c4392bfda8a4a20874ae9: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 07:59:11.277334 10812 tablet.cc:1722] T dbbae25711d94ed29223004d019ef377 P 83f20e84af4c40d09a4804b6b26e697f: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 07:59:11.280910 6172 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 0bc706e85ce74cef97a4827ea4d4aa65: rebalancer_dynamic_table_new_name [id=2c07df4ff4ae47538ff574e5adb1fd0b] alter complete (version 3)
I20240501 07:59:11.282328 6577 consensus_queue.cc:237] T 6e91683ff22243c1b3877733d74e2886 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: true } }
I20240501 07:59:11.283684 6313 raft_consensus.cc:1261] T 6e91683ff22243c1b3877733d74e2886 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Refusing update from remote peer 3b164cbbc00f4e8eb3254ddd349a3ff9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:59:11.283758 6708 raft_consensus.cc:1261] T 6e91683ff22243c1b3877733d74e2886 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Refusing update from remote peer 3b164cbbc00f4e8eb3254ddd349a3ff9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:59:11.284071 8680 consensus_queue.cc:1035] T 6e91683ff22243c1b3877733d74e2886 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 07:59:11.284247 6510 consensus_peers.cc:485] T 6e91683ff22243c1b3877733d74e2886 P 3b164cbbc00f4e8eb3254ddd349a3ff9 -> Peer 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133:39859): Couldn't send request to peer 74077b61d6f54342bf9736b4a9f2eade. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6e91683ff22243c1b3877733d74e2886. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:11.284313 9507 consensus_queue.cc:1035] T 6e91683ff22243c1b3877733d74e2886 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:11.285828 8680 raft_consensus.cc:2932] T 6e91683ff22243c1b3877733d74e2886 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: true } } }
I20240501 07:59:11.286298 6311 raft_consensus.cc:2932] T 6e91683ff22243c1b3877733d74e2886 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: true } } }
I20240501 07:59:11.288637 6168 catalog_manager.cc:5484] T 6e91683ff22243c1b3877733d74e2886 P 3b164cbbc00f4e8eb3254ddd349a3ff9 reported cstate change: config changed from index -1 to 4, NON_VOTER 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 07:59:11.289255 6708 raft_consensus.cc:2932] T 6e91683ff22243c1b3877733d74e2886 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: true } } }
I20240501 07:59:11.297780 6444 consensus_queue.cc:237] T 782e37a4ebb34a68b461a74115ae8703 P 8315d22c56f7439a9e1fc82f153f018c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: true } }
I20240501 07:59:11.299189 9056 raft_consensus.cc:1261] T 782e37a4ebb34a68b461a74115ae8703 P a505e7cb133c4392bfda8a4a20874ae9 [term 1 FOLLOWER]: Refusing update from remote peer 8315d22c56f7439a9e1fc82f153f018c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240501 07:59:11.299338 6380 consensus_peers.cc:485] T 782e37a4ebb34a68b461a74115ae8703 P 8315d22c56f7439a9e1fc82f153f018c -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 782e37a4ebb34a68b461a74115ae8703. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:11.299588 10111 consensus_queue.cc:1035] T 782e37a4ebb34a68b461a74115ae8703 P 8315d22c56f7439a9e1fc82f153f018c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 07:59:11.299894 6311 raft_consensus.cc:1261] T 782e37a4ebb34a68b461a74115ae8703 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Refusing update from remote peer 8315d22c56f7439a9e1fc82f153f018c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 07:59:11.300338 10111 consensus_queue.cc:1035] T 782e37a4ebb34a68b461a74115ae8703 P 8315d22c56f7439a9e1fc82f153f018c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 07:59:11.301371 9116 raft_consensus.cc:2932] T 782e37a4ebb34a68b461a74115ae8703 P 8315d22c56f7439a9e1fc82f153f018c [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: true } } }
I20240501 07:59:11.301831 9056 raft_consensus.cc:2932] T 782e37a4ebb34a68b461a74115ae8703 P a505e7cb133c4392bfda8a4a20874ae9 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: true } } }
I20240501 07:59:11.302628 6311 raft_consensus.cc:2932] T 782e37a4ebb34a68b461a74115ae8703 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: true } } }
I20240501 07:59:11.305332 6171 catalog_manager.cc:5484] T 782e37a4ebb34a68b461a74115ae8703 P 8b8b3ad3aa8f4053b2167dd550f2086f reported cstate change: config changed from index 6 to 7, NON_VOTER 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "8315d22c56f7439a9e1fc82f153f018c" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: true } } }
I20240501 07:59:11.340446 9056 raft_consensus.cc:1203] T 6c3c0d6fe41f4ae196ae29beb9b68125 P a505e7cb133c4392bfda8a4a20874ae9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 07:59:11.355574 6170 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55950:
table { table_name: "rebalance_test_table_14" } modify_external_catalogs: true
I20240501 07:59:11.355746 6170 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55950:
table { table_name: "rebalance_test_table_14" } modify_external_catalogs: true
I20240501 07:59:11.358331 6170 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 0bc706e85ce74cef97a4827ea4d4aa65: Sending DeleteTablet for 3 replicas of tablet 8f56a28665f342c5a77baf86dbe7d041
I20240501 07:59:11.358688 6170 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 0bc706e85ce74cef97a4827ea4d4aa65: Sending DeleteTablet for 3 replicas of tablet 3c0ec93103d349b5b57853f4b4e9f8db
I20240501 07:59:11.358963 6170 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 0bc706e85ce74cef97a4827ea4d4aa65: Sending DeleteTablet for 3 replicas of tablet b914638d3a884ae4932316bdbd1fb4a4
I20240501 07:59:11.359124 6425 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f56a28665f342c5a77baf86dbe7d041 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:11 UTC) from {username='slave'} at 127.0.0.1:45034
I20240501 07:59:11.359215 6294 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f56a28665f342c5a77baf86dbe7d041 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:11 UTC) from {username='slave'} at 127.0.0.1:45480
I20240501 07:59:11.359402 6687 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f56a28665f342c5a77baf86dbe7d041 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:11 UTC) from {username='slave'} at 127.0.0.1:38524
I20240501 07:59:11.359582 6556 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c0ec93103d349b5b57853f4b4e9f8db with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:11 UTC) from {username='slave'} at 127.0.0.1:53874
I20240501 07:59:11.359597 6295 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c0ec93103d349b5b57853f4b4e9f8db with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:11 UTC) from {username='slave'} at 127.0.0.1:45480
I20240501 07:59:11.359674 10751 tablet_replica.cc:330] T 8f56a28665f342c5a77baf86dbe7d041 P 8315d22c56f7439a9e1fc82f153f018c: stopping tablet replica
I20240501 07:59:11.359761 10753 tablet_replica.cc:330] T 3c0ec93103d349b5b57853f4b4e9f8db P 3b164cbbc00f4e8eb3254ddd349a3ff9: stopping tablet replica
I20240501 07:59:11.359828 10751 raft_consensus.cc:2229] T 8f56a28665f342c5a77baf86dbe7d041 P 8315d22c56f7439a9e1fc82f153f018c [term 1 LEADER]: Raft consensus shutting down.
I20240501 07:59:11.359885 10753 raft_consensus.cc:2229] T 3c0ec93103d349b5b57853f4b4e9f8db P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 LEADER]: Raft consensus shutting down.
I20240501 07:59:11.360049 10751 raft_consensus.cc:2258] T 8f56a28665f342c5a77baf86dbe7d041 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:11.360071 10753 raft_consensus.cc:2258] T 3c0ec93103d349b5b57853f4b4e9f8db P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:11.360494 6294 tablet_service.cc:1514] Processing DeleteTablet for tablet b914638d3a884ae4932316bdbd1fb4a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:11 UTC) from {username='slave'} at 127.0.0.1:45480
I20240501 07:59:11.360519 6687 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c0ec93103d349b5b57853f4b4e9f8db with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:11 UTC) from {username='slave'} at 127.0.0.1:38524
I20240501 07:59:11.360558 10753 ts_tablet_manager.cc:1875] T 3c0ec93103d349b5b57853f4b4e9f8db P 3b164cbbc00f4e8eb3254ddd349a3ff9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:59:11.360625 10751 ts_tablet_manager.cc:1875] T 8f56a28665f342c5a77baf86dbe7d041 P 8315d22c56f7439a9e1fc82f153f018c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:59:11.362303 10309 tablet_replica.cc:330] T 8f56a28665f342c5a77baf86dbe7d041 P 8b8b3ad3aa8f4053b2167dd550f2086f: stopping tablet replica
I20240501 07:59:11.362432 10309 raft_consensus.cc:2229] T 8f56a28665f342c5a77baf86dbe7d041 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:11.362525 10753 ts_tablet_manager.cc:1888] T 3c0ec93103d349b5b57853f4b4e9f8db P 3b164cbbc00f4e8eb3254ddd349a3ff9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 07:59:11.362576 10753 log.cc:1196] T 3c0ec93103d349b5b57853f4b4e9f8db P 3b164cbbc00f4e8eb3254ddd349a3ff9: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/3c0ec93103d349b5b57853f4b4e9f8db
I20240501 07:59:11.362581 10309 raft_consensus.cc:2258] T 8f56a28665f342c5a77baf86dbe7d041 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:11.362586 10751 ts_tablet_manager.cc:1888] T 8f56a28665f342c5a77baf86dbe7d041 P 8315d22c56f7439a9e1fc82f153f018c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 07:59:11.362643 10751 log.cc:1196] T 8f56a28665f342c5a77baf86dbe7d041 P 8315d22c56f7439a9e1fc82f153f018c: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/8f56a28665f342c5a77baf86dbe7d041
I20240501 07:59:11.362769 10753 ts_tablet_manager.cc:1909] T 3c0ec93103d349b5b57853f4b4e9f8db P 3b164cbbc00f4e8eb3254ddd349a3ff9: Deleting consensus metadata
I20240501 07:59:11.362931 10751 ts_tablet_manager.cc:1909] T 8f56a28665f342c5a77baf86dbe7d041 P 8315d22c56f7439a9e1fc82f153f018c: Deleting consensus metadata
I20240501 07:59:11.363137 10309 ts_tablet_manager.cc:1875] T 8f56a28665f342c5a77baf86dbe7d041 P 8b8b3ad3aa8f4053b2167dd550f2086f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:59:11.363536 6160 catalog_manager.cc:4830] TS 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131:34471): tablet 3c0ec93103d349b5b57853f4b4e9f8db (table rebalance_test_table_14 [id=6a104310158f4500bd3056343aa6eeca]) successfully deleted
I20240501 07:59:11.364861 10309 ts_tablet_manager.cc:1888] T 8f56a28665f342c5a77baf86dbe7d041 P 8b8b3ad3aa8f4053b2167dd550f2086f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 07:59:11.364921 10309 log.cc:1196] T 8f56a28665f342c5a77baf86dbe7d041 P 8b8b3ad3aa8f4053b2167dd550f2086f: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/8f56a28665f342c5a77baf86dbe7d041
I20240501 07:59:11.365175 10309 ts_tablet_manager.cc:1909] T 8f56a28665f342c5a77baf86dbe7d041 P 8b8b3ad3aa8f4053b2167dd550f2086f: Deleting consensus metadata
I20240501 07:59:11.365381 10310 tablet_replica.cc:330] T 8f56a28665f342c5a77baf86dbe7d041 P 1f9cb72be98941b5a8e504cbe34132b9: stopping tablet replica
I20240501 07:59:11.365537 10310 raft_consensus.cc:2229] T 8f56a28665f342c5a77baf86dbe7d041 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:11.365656 10310 raft_consensus.cc:2258] T 8f56a28665f342c5a77baf86dbe7d041 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:11.365796 10309 tablet_replica.cc:330] T 3c0ec93103d349b5b57853f4b4e9f8db P 8b8b3ad3aa8f4053b2167dd550f2086f: stopping tablet replica
I20240501 07:59:11.365887 6157 catalog_manager.cc:4830] TS 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129:46875): tablet 8f56a28665f342c5a77baf86dbe7d041 (table rebalance_test_table_14 [id=6a104310158f4500bd3056343aa6eeca]) successfully deleted
I20240501 07:59:11.365959 10309 raft_consensus.cc:2229] T 3c0ec93103d349b5b57853f4b4e9f8db P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:11.366099 10309 raft_consensus.cc:2258] T 3c0ec93103d349b5b57853f4b4e9f8db P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:11.366169 10310 ts_tablet_manager.cc:1875] T 8f56a28665f342c5a77baf86dbe7d041 P 1f9cb72be98941b5a8e504cbe34132b9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:59:11.366490 10309 ts_tablet_manager.cc:1875] T 3c0ec93103d349b5b57853f4b4e9f8db P 8b8b3ad3aa8f4053b2167dd550f2086f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:59:11.367954 10309 ts_tablet_manager.cc:1888] T 3c0ec93103d349b5b57853f4b4e9f8db P 8b8b3ad3aa8f4053b2167dd550f2086f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 07:59:11.368013 10309 log.cc:1196] T 3c0ec93103d349b5b57853f4b4e9f8db P 8b8b3ad3aa8f4053b2167dd550f2086f: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/3c0ec93103d349b5b57853f4b4e9f8db
I20240501 07:59:11.368026 10310 ts_tablet_manager.cc:1888] T 8f56a28665f342c5a77baf86dbe7d041 P 1f9cb72be98941b5a8e504cbe34132b9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 07:59:11.368105 10310 log.cc:1196] T 8f56a28665f342c5a77baf86dbe7d041 P 1f9cb72be98941b5a8e504cbe34132b9: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/8f56a28665f342c5a77baf86dbe7d041
I20240501 07:59:11.368297 10309 ts_tablet_manager.cc:1909] T 3c0ec93103d349b5b57853f4b4e9f8db P 8b8b3ad3aa8f4053b2167dd550f2086f: Deleting consensus metadata
I20240501 07:59:11.368381 10310 ts_tablet_manager.cc:1909] T 8f56a28665f342c5a77baf86dbe7d041 P 1f9cb72be98941b5a8e504cbe34132b9: Deleting consensus metadata
I20240501 07:59:11.368813 10309 tablet_replica.cc:330] T b914638d3a884ae4932316bdbd1fb4a4 P 8b8b3ad3aa8f4053b2167dd550f2086f: stopping tablet replica
I20240501 07:59:11.368933 6157 catalog_manager.cc:4830] TS 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129:46875): tablet 3c0ec93103d349b5b57853f4b4e9f8db (table rebalance_test_table_14 [id=6a104310158f4500bd3056343aa6eeca]) successfully deleted
I20240501 07:59:11.368988 10309 raft_consensus.cc:2229] T b914638d3a884ae4932316bdbd1fb4a4 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:11.369102 10309 raft_consensus.cc:2258] T b914638d3a884ae4932316bdbd1fb4a4 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:11.369202 6159 catalog_manager.cc:4830] TS 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132:36773): tablet 8f56a28665f342c5a77baf86dbe7d041 (table rebalance_test_table_14 [id=6a104310158f4500bd3056343aa6eeca]) successfully deleted
I20240501 07:59:11.369330 6159 catalog_manager.cc:4830] TS 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130:32941): tablet 8f56a28665f342c5a77baf86dbe7d041 (table rebalance_test_table_14 [id=6a104310158f4500bd3056343aa6eeca]) successfully deleted
I20240501 07:59:11.368944 10310 tablet_replica.cc:330] T 3c0ec93103d349b5b57853f4b4e9f8db P 1f9cb72be98941b5a8e504cbe34132b9: stopping tablet replica
I20240501 07:59:11.369588 10309 ts_tablet_manager.cc:1875] T b914638d3a884ae4932316bdbd1fb4a4 P 8b8b3ad3aa8f4053b2167dd550f2086f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:59:11.369349 6687 tablet_service.cc:1514] Processing DeleteTablet for tablet b914638d3a884ae4932316bdbd1fb4a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:11 UTC) from {username='slave'} at 127.0.0.1:38524
I20240501 07:59:11.369652 6425 tablet_service.cc:1514] Processing DeleteTablet for tablet b914638d3a884ae4932316bdbd1fb4a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:11 UTC) from {username='slave'} at 127.0.0.1:45034
I20240501 07:59:11.369688 10310 raft_consensus.cc:2229] T 3c0ec93103d349b5b57853f4b4e9f8db P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:11.369799 10310 raft_consensus.cc:2258] T 3c0ec93103d349b5b57853f4b4e9f8db P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:11.369843 10751 tablet_replica.cc:330] T b914638d3a884ae4932316bdbd1fb4a4 P 8315d22c56f7439a9e1fc82f153f018c: stopping tablet replica
I20240501 07:59:11.369982 10751 raft_consensus.cc:2229] T b914638d3a884ae4932316bdbd1fb4a4 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:11.370121 10751 raft_consensus.cc:2258] T b914638d3a884ae4932316bdbd1fb4a4 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:11.370267 10310 ts_tablet_manager.cc:1875] T 3c0ec93103d349b5b57853f4b4e9f8db P 1f9cb72be98941b5a8e504cbe34132b9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:59:11.370623 10751 ts_tablet_manager.cc:1875] T b914638d3a884ae4932316bdbd1fb4a4 P 8315d22c56f7439a9e1fc82f153f018c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:59:11.371356 10309 ts_tablet_manager.cc:1888] T b914638d3a884ae4932316bdbd1fb4a4 P 8b8b3ad3aa8f4053b2167dd550f2086f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 07:59:11.371421 10309 log.cc:1196] T b914638d3a884ae4932316bdbd1fb4a4 P 8b8b3ad3aa8f4053b2167dd550f2086f: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/b914638d3a884ae4932316bdbd1fb4a4
I20240501 07:59:11.371678 10309 ts_tablet_manager.cc:1909] T b914638d3a884ae4932316bdbd1fb4a4 P 8b8b3ad3aa8f4053b2167dd550f2086f: Deleting consensus metadata
I20240501 07:59:11.372231 10310 ts_tablet_manager.cc:1888] T 3c0ec93103d349b5b57853f4b4e9f8db P 1f9cb72be98941b5a8e504cbe34132b9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 07:59:11.372315 10310 log.cc:1196] T 3c0ec93103d349b5b57853f4b4e9f8db P 1f9cb72be98941b5a8e504cbe34132b9: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/3c0ec93103d349b5b57853f4b4e9f8db
I20240501 07:59:11.372408 6157 catalog_manager.cc:4830] TS 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129:46875): tablet b914638d3a884ae4932316bdbd1fb4a4 (table rebalance_test_table_14 [id=6a104310158f4500bd3056343aa6eeca]) successfully deleted
I20240501 07:59:11.372571 10310 ts_tablet_manager.cc:1909] T 3c0ec93103d349b5b57853f4b4e9f8db P 1f9cb72be98941b5a8e504cbe34132b9: Deleting consensus metadata
I20240501 07:59:11.373051 10751 ts_tablet_manager.cc:1888] T b914638d3a884ae4932316bdbd1fb4a4 P 8315d22c56f7439a9e1fc82f153f018c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 07:59:11.373211 10310 tablet_replica.cc:330] T b914638d3a884ae4932316bdbd1fb4a4 P 1f9cb72be98941b5a8e504cbe34132b9: stopping tablet replica
I20240501 07:59:11.373313 6159 catalog_manager.cc:4830] TS 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132:36773): tablet 3c0ec93103d349b5b57853f4b4e9f8db (table rebalance_test_table_14 [id=6a104310158f4500bd3056343aa6eeca]) successfully deleted
I20240501 07:59:11.373349 10310 raft_consensus.cc:2229] T b914638d3a884ae4932316bdbd1fb4a4 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 LEADER]: Raft consensus shutting down.
I20240501 07:59:11.373543 10310 raft_consensus.cc:2258] T b914638d3a884ae4932316bdbd1fb4a4 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:11.373129 10751 log.cc:1196] T b914638d3a884ae4932316bdbd1fb4a4 P 8315d22c56f7439a9e1fc82f153f018c: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/b914638d3a884ae4932316bdbd1fb4a4
I20240501 07:59:11.374076 10310 ts_tablet_manager.cc:1875] T b914638d3a884ae4932316bdbd1fb4a4 P 1f9cb72be98941b5a8e504cbe34132b9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:59:11.374970 10751 ts_tablet_manager.cc:1909] T b914638d3a884ae4932316bdbd1fb4a4 P 8315d22c56f7439a9e1fc82f153f018c: Deleting consensus metadata
I20240501 07:59:11.375998 6159 catalog_manager.cc:4830] TS 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130:32941): tablet b914638d3a884ae4932316bdbd1fb4a4 (table rebalance_test_table_14 [id=6a104310158f4500bd3056343aa6eeca]) successfully deleted
I20240501 07:59:11.376438 10310 ts_tablet_manager.cc:1888] T b914638d3a884ae4932316bdbd1fb4a4 P 1f9cb72be98941b5a8e504cbe34132b9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 07:59:11.376515 10310 log.cc:1196] T b914638d3a884ae4932316bdbd1fb4a4 P 1f9cb72be98941b5a8e504cbe34132b9: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/b914638d3a884ae4932316bdbd1fb4a4
I20240501 07:59:11.376770 10310 ts_tablet_manager.cc:1909] T b914638d3a884ae4932316bdbd1fb4a4 P 1f9cb72be98941b5a8e504cbe34132b9: Deleting consensus metadata
I20240501 07:59:11.377781 6159 catalog_manager.cc:4830] TS 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132:36773): tablet b914638d3a884ae4932316bdbd1fb4a4 (table rebalance_test_table_14 [id=6a104310158f4500bd3056343aa6eeca]) successfully deleted
I20240501 07:59:11.383016 6170 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55950:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 07:59:11.383155 6170 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55950:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 07:59:11.385462 6170 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 0bc706e85ce74cef97a4827ea4d4aa65: Sending DeleteTablet for 3 replicas of tablet dbbae25711d94ed29223004d019ef377
I20240501 07:59:11.386010 9056 raft_consensus.cc:1203] T 114b6dbafa1c4c44ac42f1d2293e3125 P a505e7cb133c4392bfda8a4a20874ae9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 07:59:11.386039 6556 tablet_service.cc:1514] Processing DeleteTablet for tablet dbbae25711d94ed29223004d019ef377 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:11 UTC) from {username='slave'} at 127.0.0.1:53874
I20240501 07:59:11.386260 10753 tablet_replica.cc:330] T dbbae25711d94ed29223004d019ef377 P 3b164cbbc00f4e8eb3254ddd349a3ff9: stopping tablet replica
I20240501 07:59:11.386409 10753 raft_consensus.cc:2229] T dbbae25711d94ed29223004d019ef377 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 LEADER]: Raft consensus shutting down.
I20240501 07:59:11.386612 10753 raft_consensus.cc:2258] T dbbae25711d94ed29223004d019ef377 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:11.386734 9035 tablet_service.cc:1514] Processing DeleteTablet for tablet dbbae25711d94ed29223004d019ef377 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:11 UTC) from {username='slave'} at 127.0.0.1:44236
I20240501 07:59:11.386745 9167 tablet_service.cc:1514] Processing DeleteTablet for tablet dbbae25711d94ed29223004d019ef377 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:11 UTC) from {username='slave'} at 127.0.0.1:33594
I20240501 07:59:11.387524 10753 ts_tablet_manager.cc:1875] T dbbae25711d94ed29223004d019ef377 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:59:11.388403 10941 tablet_replica.cc:330] T dbbae25711d94ed29223004d019ef377 P 83f20e84af4c40d09a4804b6b26e697f: stopping tablet replica
I20240501 07:59:11.388562 10941 raft_consensus.cc:2229] T dbbae25711d94ed29223004d019ef377 P 83f20e84af4c40d09a4804b6b26e697f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:11.388692 10941 raft_consensus.cc:2258] T dbbae25711d94ed29223004d019ef377 P 83f20e84af4c40d09a4804b6b26e697f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:11.389271 10941 ts_tablet_manager.cc:1875] T dbbae25711d94ed29223004d019ef377 P 83f20e84af4c40d09a4804b6b26e697f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:59:11.389730 10753 ts_tablet_manager.cc:1888] T dbbae25711d94ed29223004d019ef377 P 3b164cbbc00f4e8eb3254ddd349a3ff9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:59:11.389806 10753 log.cc:1196] T dbbae25711d94ed29223004d019ef377 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/dbbae25711d94ed29223004d019ef377
I20240501 07:59:11.390079 10753 ts_tablet_manager.cc:1909] T dbbae25711d94ed29223004d019ef377 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Deleting consensus metadata
I20240501 07:59:11.390555 10940 tablet_replica.cc:330] T dbbae25711d94ed29223004d019ef377 P a505e7cb133c4392bfda8a4a20874ae9: stopping tablet replica
I20240501 07:59:11.390686 10940 raft_consensus.cc:2229] T dbbae25711d94ed29223004d019ef377 P a505e7cb133c4392bfda8a4a20874ae9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:11.390903 6160 catalog_manager.cc:4830] TS 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131:34471): tablet dbbae25711d94ed29223004d019ef377 (table rebalancer_dynamic_table_new_name [id=2c07df4ff4ae47538ff574e5adb1fd0b]) successfully deleted
I20240501 07:59:11.391009 10940 raft_consensus.cc:2258] T dbbae25711d94ed29223004d019ef377 P a505e7cb133c4392bfda8a4a20874ae9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:11.391484 9400 consensus_queue.cc:1035] T 542c9ac08f7d4bec9b15a309c2e21364 P 13d05ebdca8c4f489c597ce550dd773d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:11.391621 10941 ts_tablet_manager.cc:1888] T dbbae25711d94ed29223004d019ef377 P 83f20e84af4c40d09a4804b6b26e697f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:59:11.391690 10941 log.cc:1196] T dbbae25711d94ed29223004d019ef377 P 83f20e84af4c40d09a4804b6b26e697f: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/dbbae25711d94ed29223004d019ef377
I20240501 07:59:11.391945 10941 ts_tablet_manager.cc:1909] T dbbae25711d94ed29223004d019ef377 P 83f20e84af4c40d09a4804b6b26e697f: Deleting consensus metadata
I20240501 07:59:11.392925 6158 catalog_manager.cc:4830] TS 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): tablet dbbae25711d94ed29223004d019ef377 (table rebalancer_dynamic_table_new_name [id=2c07df4ff4ae47538ff574e5adb1fd0b]) successfully deleted
I20240501 07:59:11.393088 10940 ts_tablet_manager.cc:1875] T dbbae25711d94ed29223004d019ef377 P a505e7cb133c4392bfda8a4a20874ae9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:59:11.394920 10235 consensus_queue.cc:1035] T 542c9ac08f7d4bec9b15a309c2e21364 P 13d05ebdca8c4f489c597ce550dd773d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:11.396250 10940 ts_tablet_manager.cc:1888] T dbbae25711d94ed29223004d019ef377 P a505e7cb133c4392bfda8a4a20874ae9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:59:11.396373 10940 log.cc:1196] T dbbae25711d94ed29223004d019ef377 P a505e7cb133c4392bfda8a4a20874ae9: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/dbbae25711d94ed29223004d019ef377
I20240501 07:59:11.397027 10940 ts_tablet_manager.cc:1909] T dbbae25711d94ed29223004d019ef377 P a505e7cb133c4392bfda8a4a20874ae9: Deleting consensus metadata
I20240501 07:59:11.397789 6158 catalog_manager.cc:4830] TS a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): tablet dbbae25711d94ed29223004d019ef377 (table rebalancer_dynamic_table_new_name [id=2c07df4ff4ae47538ff574e5adb1fd0b]) successfully deleted
I20240501 07:59:11.450572 6445 consensus_queue.cc:237] T c3a9036ccfee45498bca9681518d3699 P 8315d22c56f7439a9e1fc82f153f018c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } }
I20240501 07:59:11.452041 8793 raft_consensus.cc:1261] T c3a9036ccfee45498bca9681518d3699 P 74077b61d6f54342bf9736b4a9f2eade [term 2 FOLLOWER]: Refusing update from remote peer 8315d22c56f7439a9e1fc82f153f018c: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 07:59:11.452322 6577 raft_consensus.cc:1261] T c3a9036ccfee45498bca9681518d3699 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 2 FOLLOWER]: Refusing update from remote peer 8315d22c56f7439a9e1fc82f153f018c: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 07:59:11.452826 9489 consensus_queue.cc:1035] T c3a9036ccfee45498bca9681518d3699 P 8315d22c56f7439a9e1fc82f153f018c [LEADER]: Connected to new peer: Peer: permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 07:59:11.453444 9489 consensus_queue.cc:1035] T c3a9036ccfee45498bca9681518d3699 P 8315d22c56f7439a9e1fc82f153f018c [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 07:59:11.455298 9489 raft_consensus.cc:2932] T c3a9036ccfee45498bca9681518d3699 P 8315d22c56f7439a9e1fc82f153f018c [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } } }
I20240501 07:59:11.455863 8793 raft_consensus.cc:2932] T c3a9036ccfee45498bca9681518d3699 P 74077b61d6f54342bf9736b4a9f2eade [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } } }
I20240501 07:59:11.458365 6159 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c3a9036ccfee45498bca9681518d3699 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 07:59:11.459115 6170 catalog_manager.cc:5484] T c3a9036ccfee45498bca9681518d3699 P 74077b61d6f54342bf9736b4a9f2eade reported cstate change: config changed from index 5 to 7, VOTER 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132) evicted. New cstate: current_term: 2 leader_uuid: "8315d22c56f7439a9e1fc82f153f018c" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } } }
I20240501 07:59:11.459345 6577 raft_consensus.cc:2932] T c3a9036ccfee45498bca9681518d3699 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } } }
I20240501 07:59:11.461731 6687 tablet_service.cc:1514] Processing DeleteTablet for tablet c3a9036ccfee45498bca9681518d3699 with delete_type TABLET_DATA_TOMBSTONED (TS 1f9cb72be98941b5a8e504cbe34132b9 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:38524
I20240501 07:59:11.462755 10310 tablet_replica.cc:330] T c3a9036ccfee45498bca9681518d3699 P 1f9cb72be98941b5a8e504cbe34132b9: stopping tablet replica
I20240501 07:59:11.462899 10310 raft_consensus.cc:2229] T c3a9036ccfee45498bca9681518d3699 P 1f9cb72be98941b5a8e504cbe34132b9 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:11.462997 10310 raft_consensus.cc:2258] T c3a9036ccfee45498bca9681518d3699 P 1f9cb72be98941b5a8e504cbe34132b9 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:11.463977 10310 ts_tablet_manager.cc:1875] T c3a9036ccfee45498bca9681518d3699 P 1f9cb72be98941b5a8e504cbe34132b9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:59:11.466360 10310 ts_tablet_manager.cc:1888] T c3a9036ccfee45498bca9681518d3699 P 1f9cb72be98941b5a8e504cbe34132b9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240501 07:59:11.466440 10310 log.cc:1196] T c3a9036ccfee45498bca9681518d3699 P 1f9cb72be98941b5a8e504cbe34132b9: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/c3a9036ccfee45498bca9681518d3699
I20240501 07:59:11.466926 6159 catalog_manager.cc:4830] TS 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132:36773): tablet c3a9036ccfee45498bca9681518d3699 (table rebalance_test_table_7 [id=8e242b97203b46d0840cb3b6a649e5a7]) successfully deleted
I20240501 07:59:11.488471 6707 raft_consensus.cc:1203] T 6292ff0c7e624dd7b24715b83a30fff8 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 07:59:11.489821 6164 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55950:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 07:59:11.496842 8757 tablet_service.cc:1467] Processing CreateTablet for tablet 1eae52e7f97f45b48697c624192aa14b (DEFAULT_TABLE table=rebalancer_dynamic_table [id=a57717a5df61497abe35bc282984a9ab]), partition=
I20240501 07:59:11.496982 6687 tablet_service.cc:1467] Processing CreateTablet for tablet 1eae52e7f97f45b48697c624192aa14b (DEFAULT_TABLE table=rebalancer_dynamic_table [id=a57717a5df61497abe35bc282984a9ab]), partition=
I20240501 07:59:11.497198 8757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1eae52e7f97f45b48697c624192aa14b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:11.497215 6687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1eae52e7f97f45b48697c624192aa14b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:11.499919 9167 tablet_service.cc:1467] Processing CreateTablet for tablet 1eae52e7f97f45b48697c624192aa14b (DEFAULT_TABLE table=rebalancer_dynamic_table [id=a57717a5df61497abe35bc282984a9ab]), partition=
I20240501 07:59:11.500149 9167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1eae52e7f97f45b48697c624192aa14b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:11.500133 11012 tablet_bootstrap.cc:492] T 1eae52e7f97f45b48697c624192aa14b P 1f9cb72be98941b5a8e504cbe34132b9: Bootstrap starting.
I20240501 07:59:11.501042 11012 tablet_bootstrap.cc:654] T 1eae52e7f97f45b48697c624192aa14b P 1f9cb72be98941b5a8e504cbe34132b9: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:11.502028 11012 tablet_bootstrap.cc:492] T 1eae52e7f97f45b48697c624192aa14b P 1f9cb72be98941b5a8e504cbe34132b9: No bootstrap required, opened a new log
I20240501 07:59:11.502091 11012 ts_tablet_manager.cc:1367] T 1eae52e7f97f45b48697c624192aa14b P 1f9cb72be98941b5a8e504cbe34132b9: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240501 07:59:11.502533 11014 tablet_bootstrap.cc:492] T 1eae52e7f97f45b48697c624192aa14b P 83f20e84af4c40d09a4804b6b26e697f: Bootstrap starting.
I20240501 07:59:11.503376 11011 tablet_bootstrap.cc:492] T 1eae52e7f97f45b48697c624192aa14b P 74077b61d6f54342bf9736b4a9f2eade: Bootstrap starting.
I20240501 07:59:11.503633 11014 tablet_bootstrap.cc:654] T 1eae52e7f97f45b48697c624192aa14b P 83f20e84af4c40d09a4804b6b26e697f: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:11.503851 11012 raft_consensus.cc:348] T 1eae52e7f97f45b48697c624192aa14b P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:59:11.504102 11012 raft_consensus.cc:374] T 1eae52e7f97f45b48697c624192aa14b P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:11.504168 11012 raft_consensus.cc:729] T 1eae52e7f97f45b48697c624192aa14b P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f9cb72be98941b5a8e504cbe34132b9, State: Initialized, Role: FOLLOWER
I20240501 07:59:11.504362 11012 consensus_queue.cc:260] T 1eae52e7f97f45b48697c624192aa14b P 1f9cb72be98941b5a8e504cbe34132b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:59:11.506381 11011 tablet_bootstrap.cc:654] T 1eae52e7f97f45b48697c624192aa14b P 74077b61d6f54342bf9736b4a9f2eade: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:11.504947 11014 tablet_bootstrap.cc:492] T 1eae52e7f97f45b48697c624192aa14b P 83f20e84af4c40d09a4804b6b26e697f: No bootstrap required, opened a new log
I20240501 07:59:11.507089 11014 ts_tablet_manager.cc:1367] T 1eae52e7f97f45b48697c624192aa14b P 83f20e84af4c40d09a4804b6b26e697f: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 07:59:11.507623 11014 raft_consensus.cc:348] T 1eae52e7f97f45b48697c624192aa14b P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:59:11.507759 11014 raft_consensus.cc:374] T 1eae52e7f97f45b48697c624192aa14b P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:11.507790 11014 raft_consensus.cc:729] T 1eae52e7f97f45b48697c624192aa14b P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83f20e84af4c40d09a4804b6b26e697f, State: Initialized, Role: FOLLOWER
I20240501 07:59:11.507927 11014 consensus_queue.cc:260] T 1eae52e7f97f45b48697c624192aa14b P 83f20e84af4c40d09a4804b6b26e697f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:59:11.508219 11014 ts_tablet_manager.cc:1398] T 1eae52e7f97f45b48697c624192aa14b P 83f20e84af4c40d09a4804b6b26e697f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:11.508347 11011 tablet_bootstrap.cc:492] T 1eae52e7f97f45b48697c624192aa14b P 74077b61d6f54342bf9736b4a9f2eade: No bootstrap required, opened a new log
I20240501 07:59:11.508457 11011 ts_tablet_manager.cc:1367] T 1eae52e7f97f45b48697c624192aa14b P 74077b61d6f54342bf9736b4a9f2eade: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 07:59:11.510187 11011 raft_consensus.cc:348] T 1eae52e7f97f45b48697c624192aa14b P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:59:11.510372 11011 raft_consensus.cc:374] T 1eae52e7f97f45b48697c624192aa14b P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:11.510438 11011 raft_consensus.cc:729] T 1eae52e7f97f45b48697c624192aa14b P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74077b61d6f54342bf9736b4a9f2eade, State: Initialized, Role: FOLLOWER
I20240501 07:59:11.510582 11011 consensus_queue.cc:260] T 1eae52e7f97f45b48697c624192aa14b P 74077b61d6f54342bf9736b4a9f2eade [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:59:11.510922 11011 ts_tablet_manager.cc:1398] T 1eae52e7f97f45b48697c624192aa14b P 74077b61d6f54342bf9736b4a9f2eade: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:59:11.518301 11012 ts_tablet_manager.cc:1398] T 1eae52e7f97f45b48697c624192aa14b P 1f9cb72be98941b5a8e504cbe34132b9: Time spent starting tablet: real 0.016s user 0.000s sys 0.002s
I20240501 07:59:11.529320 9056 raft_consensus.cc:1203] T 33b3834db0cd4407ae7c26e044459e87 P a505e7cb133c4392bfda8a4a20874ae9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 07:59:11.531322 9189 raft_consensus.cc:1203] T f68dbc1f65eb4f1b91a62e2de82a7676 P 83f20e84af4c40d09a4804b6b26e697f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 07:59:11.531981 8924 raft_consensus.cc:1203] T e92d6aeeb3df4ea3852c6ebbed0e07c5 P 13d05ebdca8c4f489c597ce550dd773d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 07:59:11.533915 10403 consensus_queue.cc:1035] T 72aa4dc7664b48b7a73d34feba7d44d3 P 74077b61d6f54342bf9736b4a9f2eade [LEADER]: Connected to new peer: Peer: permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:11.538959 10403 consensus_queue.cc:1035] T 72aa4dc7664b48b7a73d34feba7d44d3 P 74077b61d6f54342bf9736b4a9f2eade [LEADER]: Connected to new peer: Peer: permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:11.569165 6425 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 07:59:11.570452 9167 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 07:59:11.571411 6556 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 07:59:11.572472 8757 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 07:59:11.576098 8904 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 07:59:11.578683 6294 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 07:59:11.583875 8924 raft_consensus.cc:1203] T b284ed2cb0264ced99a4ffd90983d902 P 13d05ebdca8c4f489c597ce550dd773d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 07:59:11.600301 6687 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 07:59:11.621922 9035 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 07:59:11.632877 6164 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55950:
name: "rebalancer_extra_table_10"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 07:59:11.637554 9189 raft_consensus.cc:1203] T 2e540f389b2a4aef93a020f549d11b67 P 83f20e84af4c40d09a4804b6b26e697f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 07:59:11.639883 10403 consensus_queue.cc:1035] T ab1666f1e5204416adb435d9bbe80433 P 74077b61d6f54342bf9736b4a9f2eade [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:11.642894 9035 tablet_service.cc:1467] Processing CreateTablet for tablet 12ce6a21d06c4ba889542758be9dc62d (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=103f9d5ca0104a7ab2f2f75dead2f70f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:11.643188 9035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12ce6a21d06c4ba889542758be9dc62d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:11.643272 9167 tablet_service.cc:1467] Processing CreateTablet for tablet 1de93edea7714491ae339db8cdfec090 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=103f9d5ca0104a7ab2f2f75dead2f70f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:11.643500 9167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1de93edea7714491ae339db8cdfec090. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:11.645435 8905 tablet_service.cc:1467] Processing CreateTablet for tablet 12ce6a21d06c4ba889542758be9dc62d (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=103f9d5ca0104a7ab2f2f75dead2f70f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:11.645530 11014 tablet_bootstrap.cc:492] T 1de93edea7714491ae339db8cdfec090 P 83f20e84af4c40d09a4804b6b26e697f: Bootstrap starting.
I20240501 07:59:11.645727 8905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12ce6a21d06c4ba889542758be9dc62d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:11.645714 6425 tablet_service.cc:1467] Processing CreateTablet for tablet 1de93edea7714491ae339db8cdfec090 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=103f9d5ca0104a7ab2f2f75dead2f70f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:11.645963 6425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1de93edea7714491ae339db8cdfec090. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:11.646517 11014 tablet_bootstrap.cc:654] T 1de93edea7714491ae339db8cdfec090 P 83f20e84af4c40d09a4804b6b26e697f: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:11.647289 6294 tablet_service.cc:1467] Processing CreateTablet for tablet 86e28c076f6749ceafe612bf730bfa68 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=103f9d5ca0104a7ab2f2f75dead2f70f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:11.647567 6294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86e28c076f6749ceafe612bf730bfa68. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:11.645435 8904 tablet_service.cc:1467] Processing CreateTablet for tablet 86e28c076f6749ceafe612bf730bfa68 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=103f9d5ca0104a7ab2f2f75dead2f70f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:11.648577 8904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86e28c076f6749ceafe612bf730bfa68. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:11.642894 9036 tablet_service.cc:1467] Processing CreateTablet for tablet 1de93edea7714491ae339db8cdfec090 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=103f9d5ca0104a7ab2f2f75dead2f70f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:11.651088 9036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1de93edea7714491ae339db8cdfec090. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:11.651149 11035 tablet_bootstrap.cc:492] T 1de93edea7714491ae339db8cdfec090 P 8315d22c56f7439a9e1fc82f153f018c: Bootstrap starting.
I20240501 07:59:11.652122 11035 tablet_bootstrap.cc:654] T 1de93edea7714491ae339db8cdfec090 P 8315d22c56f7439a9e1fc82f153f018c: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:11.652707 11034 tablet_bootstrap.cc:492] T 12ce6a21d06c4ba889542758be9dc62d P 13d05ebdca8c4f489c597ce550dd773d: Bootstrap starting.
I20240501 07:59:11.653733 8774 tablet_service.cc:1467] Processing CreateTablet for tablet 12ce6a21d06c4ba889542758be9dc62d (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=103f9d5ca0104a7ab2f2f75dead2f70f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:11.653829 11034 tablet_bootstrap.cc:654] T 12ce6a21d06c4ba889542758be9dc62d P 13d05ebdca8c4f489c597ce550dd773d: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:11.654043 8774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12ce6a21d06c4ba889542758be9dc62d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:11.654351 8757 tablet_service.cc:1467] Processing CreateTablet for tablet 86e28c076f6749ceafe612bf730bfa68 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=103f9d5ca0104a7ab2f2f75dead2f70f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:11.654567 8757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86e28c076f6749ceafe612bf730bfa68. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:11.655192 11034 tablet_bootstrap.cc:492] T 12ce6a21d06c4ba889542758be9dc62d P 13d05ebdca8c4f489c597ce550dd773d: No bootstrap required, opened a new log
I20240501 07:59:11.655306 11034 ts_tablet_manager.cc:1367] T 12ce6a21d06c4ba889542758be9dc62d P 13d05ebdca8c4f489c597ce550dd773d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 07:59:11.656294 10830 ts_tablet_manager.cc:902] T aa1b8716228b4fd9af95870b7a613fb2 P 83f20e84af4c40d09a4804b6b26e697f: Initiating tablet copy from peer 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129:46875)
I20240501 07:59:11.656684 10830 tablet_copy_client.cc:308] T aa1b8716228b4fd9af95870b7a613fb2 P 83f20e84af4c40d09a4804b6b26e697f: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:46875
I20240501 07:59:11.657068 11011 tablet_bootstrap.cc:492] T 86e28c076f6749ceafe612bf730bfa68 P 74077b61d6f54342bf9736b4a9f2eade: Bootstrap starting.
I20240501 07:59:11.657159 6335 tablet_copy_service.cc:140] P 8b8b3ad3aa8f4053b2167dd550f2086f: Received BeginTabletCopySession request for tablet aa1b8716228b4fd9af95870b7a613fb2 from peer 83f20e84af4c40d09a4804b6b26e697f ({username='slave'} at 127.0.99.136:55561)
I20240501 07:59:11.657264 6335 tablet_copy_service.cc:161] P 8b8b3ad3aa8f4053b2167dd550f2086f: Beginning new tablet copy session on tablet aa1b8716228b4fd9af95870b7a613fb2 from peer 83f20e84af4c40d09a4804b6b26e697f at {username='slave'} at 127.0.99.136:55561: session id = 83f20e84af4c40d09a4804b6b26e697f-aa1b8716228b4fd9af95870b7a613fb2
I20240501 07:59:11.657956 11011 tablet_bootstrap.cc:654] T 86e28c076f6749ceafe612bf730bfa68 P 74077b61d6f54342bf9736b4a9f2eade: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:11.658125 6335 tablet_copy_source_session.cc:205] T aa1b8716228b4fd9af95870b7a613fb2 P 8b8b3ad3aa8f4053b2167dd550f2086f: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:11.659061 11034 raft_consensus.cc:348] T 12ce6a21d06c4ba889542758be9dc62d P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } }
I20240501 07:59:11.659181 11034 raft_consensus.cc:374] T 12ce6a21d06c4ba889542758be9dc62d P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:11.659209 11034 raft_consensus.cc:729] T 12ce6a21d06c4ba889542758be9dc62d P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13d05ebdca8c4f489c597ce550dd773d, State: Initialized, Role: FOLLOWER
I20240501 07:59:11.659314 11034 consensus_queue.cc:260] T 12ce6a21d06c4ba889542758be9dc62d P 13d05ebdca8c4f489c597ce550dd773d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } }
I20240501 07:59:11.659560 11034 ts_tablet_manager.cc:1398] T 12ce6a21d06c4ba889542758be9dc62d P 13d05ebdca8c4f489c597ce550dd773d: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240501 07:59:11.659713 11034 tablet_bootstrap.cc:492] T 86e28c076f6749ceafe612bf730bfa68 P 13d05ebdca8c4f489c597ce550dd773d: Bootstrap starting.
I20240501 07:59:11.660145 10528 consensus_queue.cc:1035] T ab1666f1e5204416adb435d9bbe80433 P 74077b61d6f54342bf9736b4a9f2eade [LEADER]: Connected to new peer: Peer: permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:11.660594 11034 tablet_bootstrap.cc:654] T 86e28c076f6749ceafe612bf730bfa68 P 13d05ebdca8c4f489c597ce550dd773d: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:11.661424 10830 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa1b8716228b4fd9af95870b7a613fb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:11.663717 10830 tablet_copy_client.cc:783] T aa1b8716228b4fd9af95870b7a613fb2 P 83f20e84af4c40d09a4804b6b26e697f: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:11.663853 10830 tablet_copy_client.cc:647] T aa1b8716228b4fd9af95870b7a613fb2 P 83f20e84af4c40d09a4804b6b26e697f: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:11.664078 6313 raft_consensus.cc:1203] T fb6b46072a8d46e18b53b9f8de702859 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 07:59:11.666211 11014 tablet_bootstrap.cc:492] T 1de93edea7714491ae339db8cdfec090 P 83f20e84af4c40d09a4804b6b26e697f: No bootstrap required, opened a new log
I20240501 07:59:11.666317 11014 ts_tablet_manager.cc:1367] T 1de93edea7714491ae339db8cdfec090 P 83f20e84af4c40d09a4804b6b26e697f: Time spent bootstrapping tablet: real 0.021s user 0.000s sys 0.002s
I20240501 07:59:11.666832 11014 raft_consensus.cc:348] T 1de93edea7714491ae339db8cdfec090 P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } }
I20240501 07:59:11.666951 11014 raft_consensus.cc:374] T 1de93edea7714491ae339db8cdfec090 P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:11.666985 11014 raft_consensus.cc:729] T 1de93edea7714491ae339db8cdfec090 P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83f20e84af4c40d09a4804b6b26e697f, State: Initialized, Role: FOLLOWER
I20240501 07:59:11.667107 11014 consensus_queue.cc:260] T 1de93edea7714491ae339db8cdfec090 P 83f20e84af4c40d09a4804b6b26e697f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } }
I20240501 07:59:11.667409 11014 ts_tablet_manager.cc:1398] T 1de93edea7714491ae339db8cdfec090 P 83f20e84af4c40d09a4804b6b26e697f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:59:11.669613 11037 tablet_bootstrap.cc:492] T 1de93edea7714491ae339db8cdfec090 P a505e7cb133c4392bfda8a4a20874ae9: Bootstrap starting.
I20240501 07:59:11.670042 11011 tablet_bootstrap.cc:492] T 86e28c076f6749ceafe612bf730bfa68 P 74077b61d6f54342bf9736b4a9f2eade: No bootstrap required, opened a new log
I20240501 07:59:11.670145 11011 ts_tablet_manager.cc:1367] T 86e28c076f6749ceafe612bf730bfa68 P 74077b61d6f54342bf9736b4a9f2eade: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20240501 07:59:11.670621 11037 tablet_bootstrap.cc:654] T 1de93edea7714491ae339db8cdfec090 P a505e7cb133c4392bfda8a4a20874ae9: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:11.670617 11011 raft_consensus.cc:348] T 86e28c076f6749ceafe612bf730bfa68 P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } }
I20240501 07:59:11.670724 11011 raft_consensus.cc:374] T 86e28c076f6749ceafe612bf730bfa68 P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:11.670758 11011 raft_consensus.cc:729] T 86e28c076f6749ceafe612bf730bfa68 P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74077b61d6f54342bf9736b4a9f2eade, State: Initialized, Role: FOLLOWER
I20240501 07:59:11.670879 11011 consensus_queue.cc:260] T 86e28c076f6749ceafe612bf730bfa68 P 74077b61d6f54342bf9736b4a9f2eade [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } }
I20240501 07:59:11.671157 11011 ts_tablet_manager.cc:1398] T 86e28c076f6749ceafe612bf730bfa68 P 74077b61d6f54342bf9736b4a9f2eade: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:11.671340 11011 tablet_bootstrap.cc:492] T 12ce6a21d06c4ba889542758be9dc62d P 74077b61d6f54342bf9736b4a9f2eade: Bootstrap starting.
I20240501 07:59:11.671460 11034 tablet_bootstrap.cc:492] T 86e28c076f6749ceafe612bf730bfa68 P 13d05ebdca8c4f489c597ce550dd773d: No bootstrap required, opened a new log
I20240501 07:59:11.671545 11034 ts_tablet_manager.cc:1367] T 86e28c076f6749ceafe612bf730bfa68 P 13d05ebdca8c4f489c597ce550dd773d: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.002s
I20240501 07:59:11.671940 11034 raft_consensus.cc:348] T 86e28c076f6749ceafe612bf730bfa68 P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } }
I20240501 07:59:11.672032 11034 raft_consensus.cc:374] T 86e28c076f6749ceafe612bf730bfa68 P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:11.672061 11034 raft_consensus.cc:729] T 86e28c076f6749ceafe612bf730bfa68 P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13d05ebdca8c4f489c597ce550dd773d, State: Initialized, Role: FOLLOWER
I20240501 07:59:11.672199 11034 consensus_queue.cc:260] T 86e28c076f6749ceafe612bf730bfa68 P 13d05ebdca8c4f489c597ce550dd773d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } }
I20240501 07:59:11.672348 11011 tablet_bootstrap.cc:654] T 12ce6a21d06c4ba889542758be9dc62d P 74077b61d6f54342bf9736b4a9f2eade: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:11.672430 11034 ts_tablet_manager.cc:1398] T 86e28c076f6749ceafe612bf730bfa68 P 13d05ebdca8c4f489c597ce550dd773d: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 07:59:11.672999 11035 tablet_bootstrap.cc:492] T 1de93edea7714491ae339db8cdfec090 P 8315d22c56f7439a9e1fc82f153f018c: No bootstrap required, opened a new log
I20240501 07:59:11.673115 11035 ts_tablet_manager.cc:1367] T 1de93edea7714491ae339db8cdfec090 P 8315d22c56f7439a9e1fc82f153f018c: Time spent bootstrapping tablet: real 0.022s user 0.002s sys 0.000s
I20240501 07:59:11.673604 11035 raft_consensus.cc:348] T 1de93edea7714491ae339db8cdfec090 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } }
I20240501 07:59:11.673733 11035 raft_consensus.cc:374] T 1de93edea7714491ae339db8cdfec090 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:11.673765 11035 raft_consensus.cc:729] T 1de93edea7714491ae339db8cdfec090 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8315d22c56f7439a9e1fc82f153f018c, State: Initialized, Role: FOLLOWER
I20240501 07:59:11.673890 11035 consensus_queue.cc:260] T 1de93edea7714491ae339db8cdfec090 P 8315d22c56f7439a9e1fc82f153f018c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } }
I20240501 07:59:11.674173 11035 ts_tablet_manager.cc:1398] T 1de93edea7714491ae339db8cdfec090 P 8315d22c56f7439a9e1fc82f153f018c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:11.675760 10830 tablet_copy_client.cc:523] T aa1b8716228b4fd9af95870b7a613fb2 P 83f20e84af4c40d09a4804b6b26e697f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:11.676431 11037 tablet_bootstrap.cc:492] T 1de93edea7714491ae339db8cdfec090 P a505e7cb133c4392bfda8a4a20874ae9: No bootstrap required, opened a new log
I20240501 07:59:11.676525 11037 ts_tablet_manager.cc:1367] T 1de93edea7714491ae339db8cdfec090 P a505e7cb133c4392bfda8a4a20874ae9: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240501 07:59:11.677495 10830 tablet_bootstrap.cc:492] T aa1b8716228b4fd9af95870b7a613fb2 P 83f20e84af4c40d09a4804b6b26e697f: Bootstrap starting.
I20240501 07:59:11.678241 11037 raft_consensus.cc:348] T 1de93edea7714491ae339db8cdfec090 P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } }
I20240501 07:59:11.678376 11037 raft_consensus.cc:374] T 1de93edea7714491ae339db8cdfec090 P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:11.678418 11037 raft_consensus.cc:729] T 1de93edea7714491ae339db8cdfec090 P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a505e7cb133c4392bfda8a4a20874ae9, State: Initialized, Role: FOLLOWER
I20240501 07:59:11.678556 11037 consensus_queue.cc:260] T 1de93edea7714491ae339db8cdfec090 P a505e7cb133c4392bfda8a4a20874ae9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } }
I20240501 07:59:11.678958 11037 ts_tablet_manager.cc:1398] T 1de93edea7714491ae339db8cdfec090 P a505e7cb133c4392bfda8a4a20874ae9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 07:59:11.681722 11037 tablet_bootstrap.cc:492] T 12ce6a21d06c4ba889542758be9dc62d P a505e7cb133c4392bfda8a4a20874ae9: Bootstrap starting.
I20240501 07:59:11.683351 11037 tablet_bootstrap.cc:654] T 12ce6a21d06c4ba889542758be9dc62d P a505e7cb133c4392bfda8a4a20874ae9: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:11.684895 11037 tablet_bootstrap.cc:492] T 12ce6a21d06c4ba889542758be9dc62d P a505e7cb133c4392bfda8a4a20874ae9: No bootstrap required, opened a new log
I20240501 07:59:11.685003 11037 ts_tablet_manager.cc:1367] T 12ce6a21d06c4ba889542758be9dc62d P a505e7cb133c4392bfda8a4a20874ae9: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20240501 07:59:11.685387 11037 raft_consensus.cc:348] T 12ce6a21d06c4ba889542758be9dc62d P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } }
I20240501 07:59:11.685487 11037 raft_consensus.cc:374] T 12ce6a21d06c4ba889542758be9dc62d P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:11.685523 11037 raft_consensus.cc:729] T 12ce6a21d06c4ba889542758be9dc62d P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a505e7cb133c4392bfda8a4a20874ae9, State: Initialized, Role: FOLLOWER
I20240501 07:59:11.685642 11037 consensus_queue.cc:260] T 12ce6a21d06c4ba889542758be9dc62d P a505e7cb133c4392bfda8a4a20874ae9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } }
I20240501 07:59:11.685906 11037 ts_tablet_manager.cc:1398] T 12ce6a21d06c4ba889542758be9dc62d P a505e7cb133c4392bfda8a4a20874ae9: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:59:11.691685 11048 tablet_bootstrap.cc:492] T 86e28c076f6749ceafe612bf730bfa68 P 8b8b3ad3aa8f4053b2167dd550f2086f: Bootstrap starting.
I20240501 07:59:11.692665 11048 tablet_bootstrap.cc:654] T 86e28c076f6749ceafe612bf730bfa68 P 8b8b3ad3aa8f4053b2167dd550f2086f: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:11.693917 11048 tablet_bootstrap.cc:492] T 86e28c076f6749ceafe612bf730bfa68 P 8b8b3ad3aa8f4053b2167dd550f2086f: No bootstrap required, opened a new log
I20240501 07:59:11.694005 11048 ts_tablet_manager.cc:1367] T 86e28c076f6749ceafe612bf730bfa68 P 8b8b3ad3aa8f4053b2167dd550f2086f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 07:59:11.694553 11048 raft_consensus.cc:348] T 86e28c076f6749ceafe612bf730bfa68 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } }
I20240501 07:59:11.694685 11048 raft_consensus.cc:374] T 86e28c076f6749ceafe612bf730bfa68 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:11.694718 11048 raft_consensus.cc:729] T 86e28c076f6749ceafe612bf730bfa68 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b8b3ad3aa8f4053b2167dd550f2086f, State: Initialized, Role: FOLLOWER
I20240501 07:59:11.694840 11048 consensus_queue.cc:260] T 86e28c076f6749ceafe612bf730bfa68 P 8b8b3ad3aa8f4053b2167dd550f2086f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } }
I20240501 07:59:11.695112 11048 ts_tablet_manager.cc:1398] T 86e28c076f6749ceafe612bf730bfa68 P 8b8b3ad3aa8f4053b2167dd550f2086f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:11.696483 11011 tablet_bootstrap.cc:492] T 12ce6a21d06c4ba889542758be9dc62d P 74077b61d6f54342bf9736b4a9f2eade: No bootstrap required, opened a new log
I20240501 07:59:11.696575 11011 ts_tablet_manager.cc:1367] T 12ce6a21d06c4ba889542758be9dc62d P 74077b61d6f54342bf9736b4a9f2eade: Time spent bootstrapping tablet: real 0.025s user 0.001s sys 0.002s
I20240501 07:59:11.697254 11011 raft_consensus.cc:348] T 12ce6a21d06c4ba889542758be9dc62d P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } }
I20240501 07:59:11.697360 11011 raft_consensus.cc:374] T 12ce6a21d06c4ba889542758be9dc62d P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:11.697396 11011 raft_consensus.cc:729] T 12ce6a21d06c4ba889542758be9dc62d P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74077b61d6f54342bf9736b4a9f2eade, State: Initialized, Role: FOLLOWER
I20240501 07:59:11.697502 11011 consensus_queue.cc:260] T 12ce6a21d06c4ba889542758be9dc62d P 74077b61d6f54342bf9736b4a9f2eade [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } }
I20240501 07:59:11.697770 11011 ts_tablet_manager.cc:1398] T 12ce6a21d06c4ba889542758be9dc62d P 74077b61d6f54342bf9736b4a9f2eade: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 07:59:11.708464 10830 tablet_bootstrap.cc:492] T aa1b8716228b4fd9af95870b7a613fb2 P 83f20e84af4c40d09a4804b6b26e697f: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:11.709607 10830 tablet_bootstrap.cc:492] T aa1b8716228b4fd9af95870b7a613fb2 P 83f20e84af4c40d09a4804b6b26e697f: Bootstrap complete.
I20240501 07:59:11.709890 10830 ts_tablet_manager.cc:1367] T aa1b8716228b4fd9af95870b7a613fb2 P 83f20e84af4c40d09a4804b6b26e697f: Time spent bootstrapping tablet: real 0.032s user 0.027s sys 0.000s
I20240501 07:59:11.710414 10830 raft_consensus.cc:348] T aa1b8716228b4fd9af95870b7a613fb2 P 83f20e84af4c40d09a4804b6b26e697f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { replace: true } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: true } }
I20240501 07:59:11.710669 10830 raft_consensus.cc:729] T aa1b8716228b4fd9af95870b7a613fb2 P 83f20e84af4c40d09a4804b6b26e697f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 83f20e84af4c40d09a4804b6b26e697f, State: Initialized, Role: LEARNER
I20240501 07:59:11.710808 10830 consensus_queue.cc:260] T aa1b8716228b4fd9af95870b7a613fb2 P 83f20e84af4c40d09a4804b6b26e697f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { replace: true } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: true } }
I20240501 07:59:11.711786 10830 ts_tablet_manager.cc:1398] T aa1b8716228b4fd9af95870b7a613fb2 P 83f20e84af4c40d09a4804b6b26e697f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:59:11.712399 6335 tablet_copy_service.cc:342] P 8b8b3ad3aa8f4053b2167dd550f2086f: Request end of tablet copy session 83f20e84af4c40d09a4804b6b26e697f-aa1b8716228b4fd9af95870b7a613fb2 received from {username='slave'} at 127.0.99.136:55561
I20240501 07:59:11.712492 6335 tablet_copy_service.cc:433] P 8b8b3ad3aa8f4053b2167dd550f2086f: ending tablet copy session 83f20e84af4c40d09a4804b6b26e697f-aa1b8716228b4fd9af95870b7a613fb2 on tablet aa1b8716228b4fd9af95870b7a613fb2 with peer 83f20e84af4c40d09a4804b6b26e697f
I20240501 07:59:11.726532 6707 consensus_queue.cc:237] T 7dd6566ccb6e43388146a6dfd1c4a88c P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: true } }
I20240501 07:59:11.727983 6577 raft_consensus.cc:1261] T 7dd6566ccb6e43388146a6dfd1c4a88c P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Refusing update from remote peer 1f9cb72be98941b5a8e504cbe34132b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:59:11.728129 6445 raft_consensus.cc:1261] T 7dd6566ccb6e43388146a6dfd1c4a88c P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Refusing update from remote peer 1f9cb72be98941b5a8e504cbe34132b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 07:59:11.728437 6643 consensus_peers.cc:485] T 7dd6566ccb6e43388146a6dfd1c4a88c P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134:40141): Couldn't send request to peer 13d05ebdca8c4f489c597ce550dd773d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7dd6566ccb6e43388146a6dfd1c4a88c. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:11.728449 10324 consensus_queue.cc:1035] T 7dd6566ccb6e43388146a6dfd1c4a88c P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:11.728814 10324 consensus_queue.cc:1035] T 7dd6566ccb6e43388146a6dfd1c4a88c P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:11.730489 9990 raft_consensus.cc:2932] T 7dd6566ccb6e43388146a6dfd1c4a88c P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: true } } }
I20240501 07:59:11.731361 6577 raft_consensus.cc:2932] T 7dd6566ccb6e43388146a6dfd1c4a88c P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: true } } }
I20240501 07:59:11.732535 6445 raft_consensus.cc:2932] T 7dd6566ccb6e43388146a6dfd1c4a88c P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: true } } }
I20240501 07:59:11.735586 6168 catalog_manager.cc:5484] T 7dd6566ccb6e43388146a6dfd1c4a88c P 3b164cbbc00f4e8eb3254ddd349a3ff9 reported cstate change: config changed from index -1 to 4, NON_VOTER 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "1f9cb72be98941b5a8e504cbe34132b9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: true } } }
I20240501 07:59:11.742686 6707 consensus_queue.cc:237] T 8454f24b0c28451b8ee51be39ca3e07f P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: true } }
I20240501 07:59:11.744089 6313 raft_consensus.cc:1261] T 8454f24b0c28451b8ee51be39ca3e07f P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Refusing update from remote peer 1f9cb72be98941b5a8e504cbe34132b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240501 07:59:11.744195 6643 consensus_peers.cc:485] T 8454f24b0c28451b8ee51be39ca3e07f P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134:40141): Couldn't send request to peer 13d05ebdca8c4f489c597ce550dd773d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8454f24b0c28451b8ee51be39ca3e07f. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:11.744383 9056 raft_consensus.cc:1261] T 8454f24b0c28451b8ee51be39ca3e07f P a505e7cb133c4392bfda8a4a20874ae9 [term 1 FOLLOWER]: Refusing update from remote peer 1f9cb72be98941b5a8e504cbe34132b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 07:59:11.744560 9618 consensus_queue.cc:1035] T 8454f24b0c28451b8ee51be39ca3e07f P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 07:59:11.744896 9618 consensus_queue.cc:1035] T 8454f24b0c28451b8ee51be39ca3e07f P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 07:59:11.746451 9618 raft_consensus.cc:2932] T 8454f24b0c28451b8ee51be39ca3e07f P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: true } } }
I20240501 07:59:11.747231 9056 raft_consensus.cc:2932] T 8454f24b0c28451b8ee51be39ca3e07f P a505e7cb133c4392bfda8a4a20874ae9 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: true } } }
I20240501 07:59:11.748013 6313 raft_consensus.cc:2932] T 8454f24b0c28451b8ee51be39ca3e07f P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: true } } }
I20240501 07:59:11.747227 10945 raft_consensus.cc:482] T 1de93edea7714491ae339db8cdfec090 P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:11.749648 10945 raft_consensus.cc:504] T 1de93edea7714491ae339db8cdfec090 P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } }
I20240501 07:59:11.750126 10945 leader_election.cc:288] T 1de93edea7714491ae339db8cdfec090 P a505e7cb133c4392bfda8a4a20874ae9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149), 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130:32941)
I20240501 07:59:11.750798 9189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1de93edea7714491ae339db8cdfec090" candidate_uuid: "a505e7cb133c4392bfda8a4a20874ae9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83f20e84af4c40d09a4804b6b26e697f" is_pre_election: true
I20240501 07:59:11.750774 6165 catalog_manager.cc:5484] T 8454f24b0c28451b8ee51be39ca3e07f P a505e7cb133c4392bfda8a4a20874ae9 reported cstate change: config changed from index 6 to 7, NON_VOTER 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "1f9cb72be98941b5a8e504cbe34132b9" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: true } } }
I20240501 07:59:11.750936 9189 raft_consensus.cc:2454] T 1de93edea7714491ae339db8cdfec090 P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a505e7cb133c4392bfda8a4a20874ae9 in term 0.
I20240501 07:59:11.751232 8990 leader_election.cc:302] T 1de93edea7714491ae339db8cdfec090 P a505e7cb133c4392bfda8a4a20874ae9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83f20e84af4c40d09a4804b6b26e697f, a505e7cb133c4392bfda8a4a20874ae9; no voters:
I20240501 07:59:11.751381 10945 raft_consensus.cc:2781] T 1de93edea7714491ae339db8cdfec090 P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:11.751436 10945 raft_consensus.cc:482] T 1de93edea7714491ae339db8cdfec090 P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:11.751392 6445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1de93edea7714491ae339db8cdfec090" candidate_uuid: "a505e7cb133c4392bfda8a4a20874ae9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8315d22c56f7439a9e1fc82f153f018c" is_pre_election: true
I20240501 07:59:11.751477 10945 raft_consensus.cc:3037] T 1de93edea7714491ae339db8cdfec090 P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:11.751538 6445 raft_consensus.cc:2454] T 1de93edea7714491ae339db8cdfec090 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a505e7cb133c4392bfda8a4a20874ae9 in term 0.
I20240501 07:59:11.752442 10945 raft_consensus.cc:504] T 1de93edea7714491ae339db8cdfec090 P a505e7cb133c4392bfda8a4a20874ae9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } }
I20240501 07:59:11.752794 10945 leader_election.cc:288] T 1de93edea7714491ae339db8cdfec090 P a505e7cb133c4392bfda8a4a20874ae9 [CANDIDATE]: Term 1 election: Requested vote from peers 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149), 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130:32941)
I20240501 07:59:11.753264 6445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1de93edea7714491ae339db8cdfec090" candidate_uuid: "a505e7cb133c4392bfda8a4a20874ae9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8315d22c56f7439a9e1fc82f153f018c"
I20240501 07:59:11.753371 6445 raft_consensus.cc:3037] T 1de93edea7714491ae339db8cdfec090 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:11.753422 9189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1de93edea7714491ae339db8cdfec090" candidate_uuid: "a505e7cb133c4392bfda8a4a20874ae9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83f20e84af4c40d09a4804b6b26e697f"
I20240501 07:59:11.753530 9189 raft_consensus.cc:3037] T 1de93edea7714491ae339db8cdfec090 P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:11.754403 6445 raft_consensus.cc:2454] T 1de93edea7714491ae339db8cdfec090 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a505e7cb133c4392bfda8a4a20874ae9 in term 1.
I20240501 07:59:11.754559 9189 raft_consensus.cc:2454] T 1de93edea7714491ae339db8cdfec090 P 83f20e84af4c40d09a4804b6b26e697f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a505e7cb133c4392bfda8a4a20874ae9 in term 1.
I20240501 07:59:11.755074 8991 leader_election.cc:302] T 1de93edea7714491ae339db8cdfec090 P a505e7cb133c4392bfda8a4a20874ae9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8315d22c56f7439a9e1fc82f153f018c, a505e7cb133c4392bfda8a4a20874ae9; no voters:
I20240501 07:59:11.755242 10945 raft_consensus.cc:2781] T 1de93edea7714491ae339db8cdfec090 P a505e7cb133c4392bfda8a4a20874ae9 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:11.755335 10945 raft_consensus.cc:686] T 1de93edea7714491ae339db8cdfec090 P a505e7cb133c4392bfda8a4a20874ae9 [term 1 LEADER]: Becoming Leader. State: Replica: a505e7cb133c4392bfda8a4a20874ae9, State: Running, Role: LEADER
I20240501 07:59:11.755472 10945 consensus_queue.cc:237] T 1de93edea7714491ae339db8cdfec090 P a505e7cb133c4392bfda8a4a20874ae9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } }
I20240501 07:59:11.757405 6168 catalog_manager.cc:5484] T 1de93edea7714491ae339db8cdfec090 P a505e7cb133c4392bfda8a4a20874ae9 reported cstate change: term changed from 0 to 1, leader changed from <none> to a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135). New cstate: current_term: 1 leader_uuid: "a505e7cb133c4392bfda8a4a20874ae9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } health_report { overall_health: UNKNOWN } } }
I20240501 07:59:11.762547 6707 consensus_queue.cc:237] T ae46990350284295bf14aa64d4b6c93e P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: true } }
I20240501 07:59:11.763829 6313 raft_consensus.cc:1261] T ae46990350284295bf14aa64d4b6c93e P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Refusing update from remote peer 1f9cb72be98941b5a8e504cbe34132b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:59:11.763833 6445 raft_consensus.cc:1261] T ae46990350284295bf14aa64d4b6c93e P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Refusing update from remote peer 1f9cb72be98941b5a8e504cbe34132b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 07:59:11.763957 6642 consensus_peers.cc:485] T ae46990350284295bf14aa64d4b6c93e P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ae46990350284295bf14aa64d4b6c93e. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:11.764218 9618 consensus_queue.cc:1035] T ae46990350284295bf14aa64d4b6c93e P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:11.765336 9618 consensus_queue.cc:1035] T ae46990350284295bf14aa64d4b6c93e P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:11.765897 9990 raft_consensus.cc:2932] T ae46990350284295bf14aa64d4b6c93e P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: true } } }
I20240501 07:59:11.766336 6445 raft_consensus.cc:2932] T ae46990350284295bf14aa64d4b6c93e P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: true } } }
I20240501 07:59:11.768657 6168 catalog_manager.cc:5484] T ae46990350284295bf14aa64d4b6c93e P 1f9cb72be98941b5a8e504cbe34132b9 reported cstate change: config changed from index -1 to 4, NON_VOTER a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "1f9cb72be98941b5a8e504cbe34132b9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 07:59:11.770098 10867 raft_consensus.cc:1050] T 6c3c0d6fe41f4ae196ae29beb9b68125 P 8b8b3ad3aa8f4053b2167dd550f2086f: attempting to promote NON_VOTER a505e7cb133c4392bfda8a4a20874ae9 to VOTER
I20240501 07:59:11.770531 10867 consensus_queue.cc:237] T 6c3c0d6fe41f4ae196ae29beb9b68125 P 8b8b3ad3aa8f4053b2167dd550f2086f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { replace: true } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false } }
I20240501 07:59:11.772008 6707 raft_consensus.cc:1261] T 6c3c0d6fe41f4ae196ae29beb9b68125 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Refusing update from remote peer 8b8b3ad3aa8f4053b2167dd550f2086f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:59:11.772362 11085 consensus_queue.cc:1035] T 6c3c0d6fe41f4ae196ae29beb9b68125 P 8b8b3ad3aa8f4053b2167dd550f2086f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:59:11.772411 6577 raft_consensus.cc:1261] T 6c3c0d6fe41f4ae196ae29beb9b68125 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Refusing update from remote peer 8b8b3ad3aa8f4053b2167dd550f2086f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:59:11.772853 11085 consensus_queue.cc:1035] T 6c3c0d6fe41f4ae196ae29beb9b68125 P 8b8b3ad3aa8f4053b2167dd550f2086f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:59:11.773249 6313 raft_consensus.cc:2932] T ae46990350284295bf14aa64d4b6c93e P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: true } } }
I20240501 07:59:11.773401 9056 raft_consensus.cc:1261] T 6c3c0d6fe41f4ae196ae29beb9b68125 P a505e7cb133c4392bfda8a4a20874ae9 [term 1 LEARNER]: Refusing update from remote peer 8b8b3ad3aa8f4053b2167dd550f2086f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:59:11.774849 11085 consensus_queue.cc:1035] T 6c3c0d6fe41f4ae196ae29beb9b68125 P 8b8b3ad3aa8f4053b2167dd550f2086f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:59:11.775180 11085 raft_consensus.cc:2932] T 6c3c0d6fe41f4ae196ae29beb9b68125 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { replace: true } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false } } }
I20240501 07:59:11.775528 6577 raft_consensus.cc:2932] T 6c3c0d6fe41f4ae196ae29beb9b68125 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { replace: true } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false } } }
I20240501 07:59:11.776115 9056 raft_consensus.cc:2932] T 6c3c0d6fe41f4ae196ae29beb9b68125 P a505e7cb133c4392bfda8a4a20874ae9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { replace: true } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false } } }
I20240501 07:59:11.778559 6707 raft_consensus.cc:2932] T 6c3c0d6fe41f4ae196ae29beb9b68125 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { replace: true } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false } } }
I20240501 07:59:11.778929 6165 catalog_manager.cc:5484] T 6c3c0d6fe41f4ae196ae29beb9b68125 P 3b164cbbc00f4e8eb3254ddd349a3ff9 reported cstate change: config changed from index 4 to 5, a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { replace: true } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false } } }
I20240501 07:59:11.785930 10830 ts_tablet_manager.cc:902] T 782e37a4ebb34a68b461a74115ae8703 P 83f20e84af4c40d09a4804b6b26e697f: Initiating tablet copy from peer 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130:32941)
I20240501 07:59:11.786433 10830 tablet_copy_client.cc:308] T 782e37a4ebb34a68b461a74115ae8703 P 83f20e84af4c40d09a4804b6b26e697f: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:32941
I20240501 07:59:11.786857 6466 tablet_copy_service.cc:140] P 8315d22c56f7439a9e1fc82f153f018c: Received BeginTabletCopySession request for tablet 782e37a4ebb34a68b461a74115ae8703 from peer 83f20e84af4c40d09a4804b6b26e697f ({username='slave'} at 127.0.99.136:51357)
I20240501 07:59:11.786976 6466 tablet_copy_service.cc:161] P 8315d22c56f7439a9e1fc82f153f018c: Beginning new tablet copy session on tablet 782e37a4ebb34a68b461a74115ae8703 from peer 83f20e84af4c40d09a4804b6b26e697f at {username='slave'} at 127.0.99.136:51357: session id = 83f20e84af4c40d09a4804b6b26e697f-782e37a4ebb34a68b461a74115ae8703
I20240501 07:59:11.787119 8793 consensus_queue.cc:237] T 72aa4dc7664b48b7a73d34feba7d44d3 P 74077b61d6f54342bf9736b4a9f2eade [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { replace: true } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { promote: true } }
I20240501 07:59:11.787976 6466 tablet_copy_source_session.cc:205] T 782e37a4ebb34a68b461a74115ae8703 P 8315d22c56f7439a9e1fc82f153f018c: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:11.788653 9056 raft_consensus.cc:1261] T 72aa4dc7664b48b7a73d34feba7d44d3 P a505e7cb133c4392bfda8a4a20874ae9 [term 1 FOLLOWER]: Refusing update from remote peer 74077b61d6f54342bf9736b4a9f2eade: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:11.788751 8924 raft_consensus.cc:1261] T 72aa4dc7664b48b7a73d34feba7d44d3 P 13d05ebdca8c4f489c597ce550dd773d [term 1 FOLLOWER]: Refusing update from remote peer 74077b61d6f54342bf9736b4a9f2eade: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 07:59:11.789000 8729 consensus_peers.cc:485] T 72aa4dc7664b48b7a73d34feba7d44d3 P 74077b61d6f54342bf9736b4a9f2eade -> Peer 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130:32941): Couldn't send request to peer 8315d22c56f7439a9e1fc82f153f018c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 72aa4dc7664b48b7a73d34feba7d44d3. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:11.789181 10069 consensus_queue.cc:1035] T 72aa4dc7664b48b7a73d34feba7d44d3 P 74077b61d6f54342bf9736b4a9f2eade [LEADER]: Connected to new peer: Peer: permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 07:59:11.789294 10528 consensus_queue.cc:1035] T 72aa4dc7664b48b7a73d34feba7d44d3 P 74077b61d6f54342bf9736b4a9f2eade [LEADER]: Connected to new peer: Peer: permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 07:59:11.789393 10830 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 782e37a4ebb34a68b461a74115ae8703. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:11.790930 10867 raft_consensus.cc:1050] T 2398430a23094711a13a86cc88e27a78 P 8b8b3ad3aa8f4053b2167dd550f2086f: attempting to promote NON_VOTER 74077b61d6f54342bf9736b4a9f2eade to VOTER
I20240501 07:59:11.791033 10069 raft_consensus.cc:2932] T 72aa4dc7664b48b7a73d34feba7d44d3 P 74077b61d6f54342bf9736b4a9f2eade [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { replace: true } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { promote: true } } }
I20240501 07:59:11.791379 10867 consensus_queue.cc:237] T 2398430a23094711a13a86cc88e27a78 P 8b8b3ad3aa8f4053b2167dd550f2086f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } }
I20240501 07:59:11.791572 10830 tablet_copy_client.cc:783] T 782e37a4ebb34a68b461a74115ae8703 P 83f20e84af4c40d09a4804b6b26e697f: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:11.791720 10830 tablet_copy_client.cc:647] T 782e37a4ebb34a68b461a74115ae8703 P 83f20e84af4c40d09a4804b6b26e697f: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:11.792867 6445 raft_consensus.cc:1261] T 2398430a23094711a13a86cc88e27a78 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Refusing update from remote peer 8b8b3ad3aa8f4053b2167dd550f2086f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:59:11.793610 6577 raft_consensus.cc:1261] T 2398430a23094711a13a86cc88e27a78 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Refusing update from remote peer 8b8b3ad3aa8f4053b2167dd550f2086f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:59:11.793494 9056 raft_consensus.cc:2932] T 72aa4dc7664b48b7a73d34feba7d44d3 P a505e7cb133c4392bfda8a4a20874ae9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { replace: true } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { promote: true } } }
I20240501 07:59:11.793900 8924 raft_consensus.cc:2932] T 72aa4dc7664b48b7a73d34feba7d44d3 P 13d05ebdca8c4f489c597ce550dd773d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { replace: true } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { promote: true } } }
I20240501 07:59:11.794021 9964 consensus_queue.cc:1035] T 2398430a23094711a13a86cc88e27a78 P 8b8b3ad3aa8f4053b2167dd550f2086f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:59:11.795212 9964 consensus_queue.cc:1035] T 2398430a23094711a13a86cc88e27a78 P 8b8b3ad3aa8f4053b2167dd550f2086f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:59:11.795523 6171 catalog_manager.cc:5484] T 72aa4dc7664b48b7a73d34feba7d44d3 P 74077b61d6f54342bf9736b4a9f2eade reported cstate change: config changed from index -1 to 2, NON_VOTER 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130) added. New cstate: current_term: 1 leader_uuid: "74077b61d6f54342bf9736b4a9f2eade" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 07:59:11.797073 8793 raft_consensus.cc:1261] T 2398430a23094711a13a86cc88e27a78 P 74077b61d6f54342bf9736b4a9f2eade [term 1 LEARNER]: Refusing update from remote peer 8b8b3ad3aa8f4053b2167dd550f2086f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:59:11.797237 11045 raft_consensus.cc:482] T 1eae52e7f97f45b48697c624192aa14b P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:11.797336 11045 raft_consensus.cc:504] T 1eae52e7f97f45b48697c624192aa14b P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:59:11.797696 11085 consensus_queue.cc:1035] T 2398430a23094711a13a86cc88e27a78 P 8b8b3ad3aa8f4053b2167dd550f2086f [LEADER]: Connected to new peer: Peer: permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:59:11.797822 11045 leader_election.cc:288] T 1eae52e7f97f45b48697c624192aa14b P 83f20e84af4c40d09a4804b6b26e697f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133:39859), 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132:36773)
I20240501 07:59:11.798386 8794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1eae52e7f97f45b48697c624192aa14b" candidate_uuid: "83f20e84af4c40d09a4804b6b26e697f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74077b61d6f54342bf9736b4a9f2eade" is_pre_election: true
I20240501 07:59:11.798502 8794 raft_consensus.cc:2454] T 1eae52e7f97f45b48697c624192aa14b P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83f20e84af4c40d09a4804b6b26e697f in term 0.
I20240501 07:59:11.798583 6707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1eae52e7f97f45b48697c624192aa14b" candidate_uuid: "83f20e84af4c40d09a4804b6b26e697f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f9cb72be98941b5a8e504cbe34132b9" is_pre_election: true
I20240501 07:59:11.798723 6707 raft_consensus.cc:2454] T 1eae52e7f97f45b48697c624192aa14b P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83f20e84af4c40d09a4804b6b26e697f in term 0.
I20240501 07:59:11.799057 9124 leader_election.cc:302] T 1eae52e7f97f45b48697c624192aa14b P 83f20e84af4c40d09a4804b6b26e697f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f9cb72be98941b5a8e504cbe34132b9, 83f20e84af4c40d09a4804b6b26e697f; no voters:
I20240501 07:59:11.799288 11045 raft_consensus.cc:2781] T 1eae52e7f97f45b48697c624192aa14b P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:11.799340 11045 raft_consensus.cc:482] T 1eae52e7f97f45b48697c624192aa14b P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:11.799381 11045 raft_consensus.cc:3037] T 1eae52e7f97f45b48697c624192aa14b P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:11.800371 11045 raft_consensus.cc:504] T 1eae52e7f97f45b48697c624192aa14b P 83f20e84af4c40d09a4804b6b26e697f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:59:11.800725 11045 leader_election.cc:288] T 1eae52e7f97f45b48697c624192aa14b P 83f20e84af4c40d09a4804b6b26e697f [CANDIDATE]: Term 1 election: Requested vote from peers 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133:39859), 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132:36773)
I20240501 07:59:11.801002 8793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1eae52e7f97f45b48697c624192aa14b" candidate_uuid: "83f20e84af4c40d09a4804b6b26e697f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74077b61d6f54342bf9736b4a9f2eade"
I20240501 07:59:11.801131 8793 raft_consensus.cc:3037] T 1eae52e7f97f45b48697c624192aa14b P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:11.801594 6707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1eae52e7f97f45b48697c624192aa14b" candidate_uuid: "83f20e84af4c40d09a4804b6b26e697f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f9cb72be98941b5a8e504cbe34132b9"
I20240501 07:59:11.801669 10830 tablet_copy_client.cc:523] T 782e37a4ebb34a68b461a74115ae8703 P 83f20e84af4c40d09a4804b6b26e697f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:11.801699 6707 raft_consensus.cc:3037] T 1eae52e7f97f45b48697c624192aa14b P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:11.802109 8793 raft_consensus.cc:2454] T 1eae52e7f97f45b48697c624192aa14b P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83f20e84af4c40d09a4804b6b26e697f in term 1.
I20240501 07:59:11.802742 6707 raft_consensus.cc:2454] T 1eae52e7f97f45b48697c624192aa14b P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83f20e84af4c40d09a4804b6b26e697f in term 1.
I20240501 07:59:11.803130 9124 leader_election.cc:302] T 1eae52e7f97f45b48697c624192aa14b P 83f20e84af4c40d09a4804b6b26e697f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f9cb72be98941b5a8e504cbe34132b9, 83f20e84af4c40d09a4804b6b26e697f; no voters:
I20240501 07:59:11.803323 11045 raft_consensus.cc:2781] T 1eae52e7f97f45b48697c624192aa14b P 83f20e84af4c40d09a4804b6b26e697f [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:11.803397 11045 raft_consensus.cc:686] T 1eae52e7f97f45b48697c624192aa14b P 83f20e84af4c40d09a4804b6b26e697f [term 1 LEADER]: Becoming Leader. State: Replica: 83f20e84af4c40d09a4804b6b26e697f, State: Running, Role: LEADER
I20240501 07:59:11.803262 9964 raft_consensus.cc:2932] T 2398430a23094711a13a86cc88e27a78 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } } }
I20240501 07:59:11.803552 11045 consensus_queue.cc:237] T 1eae52e7f97f45b48697c624192aa14b P 83f20e84af4c40d09a4804b6b26e697f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:59:11.803786 6577 raft_consensus.cc:2932] T 2398430a23094711a13a86cc88e27a78 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } } }
I20240501 07:59:11.803138 10830 tablet_bootstrap.cc:492] T 782e37a4ebb34a68b461a74115ae8703 P 83f20e84af4c40d09a4804b6b26e697f: Bootstrap starting.
I20240501 07:59:11.805315 8793 raft_consensus.cc:2932] T 2398430a23094711a13a86cc88e27a78 P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } } }
I20240501 07:59:11.805577 6445 raft_consensus.cc:2932] T 2398430a23094711a13a86cc88e27a78 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } } }
I20240501 07:59:11.807382 6172 catalog_manager.cc:5484] T 2398430a23094711a13a86cc88e27a78 P 3b164cbbc00f4e8eb3254ddd349a3ff9 reported cstate change: config changed from index 4 to 5, 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } } }
I20240501 07:59:11.810009 6164 catalog_manager.cc:5484] T 1eae52e7f97f45b48697c624192aa14b P 83f20e84af4c40d09a4804b6b26e697f reported cstate change: term changed from 0 to 1, leader changed from <none> to 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136). New cstate: current_term: 1 leader_uuid: "83f20e84af4c40d09a4804b6b26e697f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } health_report { overall_health: UNKNOWN } } }
I20240501 07:59:11.813231 6313 consensus_queue.cc:237] T 2398430a23094711a13a86cc88e27a78 P 8b8b3ad3aa8f4053b2167dd550f2086f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } }
I20240501 07:59:11.814491 8793 raft_consensus.cc:1261] T 2398430a23094711a13a86cc88e27a78 P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Refusing update from remote peer 8b8b3ad3aa8f4053b2167dd550f2086f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 07:59:11.814800 6445 raft_consensus.cc:1261] T 2398430a23094711a13a86cc88e27a78 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Refusing update from remote peer 8b8b3ad3aa8f4053b2167dd550f2086f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 07:59:11.815105 9373 consensus_queue.cc:1035] T 2398430a23094711a13a86cc88e27a78 P 8b8b3ad3aa8f4053b2167dd550f2086f [LEADER]: Connected to new peer: Peer: permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 07:59:11.815287 11085 consensus_queue.cc:1035] T 2398430a23094711a13a86cc88e27a78 P 8b8b3ad3aa8f4053b2167dd550f2086f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 07:59:11.817171 10046 raft_consensus.cc:1050] T 114b6dbafa1c4c44ac42f1d2293e3125 P 3b164cbbc00f4e8eb3254ddd349a3ff9: attempting to promote NON_VOTER a505e7cb133c4392bfda8a4a20874ae9 to VOTER
I20240501 07:59:11.817564 10046 consensus_queue.cc:237] T 114b6dbafa1c4c44ac42f1d2293e3125 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false } }
I20240501 07:59:11.817601 9189 consensus_queue.cc:237] T 2976d6353e8947b7b6c37f72efbeeac4 P 83f20e84af4c40d09a4804b6b26e697f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: true } }
I20240501 07:59:11.819121 9056 raft_consensus.cc:1261] T 2976d6353e8947b7b6c37f72efbeeac4 P a505e7cb133c4392bfda8a4a20874ae9 [term 1 FOLLOWER]: Refusing update from remote peer 83f20e84af4c40d09a4804b6b26e697f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:11.819228 11085 raft_consensus.cc:2932] T 2398430a23094711a13a86cc88e27a78 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } } }
I20240501 07:59:11.819459 9055 raft_consensus.cc:1261] T 114b6dbafa1c4c44ac42f1d2293e3125 P a505e7cb133c4392bfda8a4a20874ae9 [term 1 LEARNER]: Refusing update from remote peer 3b164cbbc00f4e8eb3254ddd349a3ff9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:59:11.820351 9507 consensus_queue.cc:1035] T 114b6dbafa1c4c44ac42f1d2293e3125 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:59:11.821604 11110 ts_tablet_manager.cc:902] T 6e91683ff22243c1b3877733d74e2886 P 74077b61d6f54342bf9736b4a9f2eade: Initiating tablet copy from peer 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131:34471)
I20240501 07:59:11.821693 10848 consensus_queue.cc:1035] T 2976d6353e8947b7b6c37f72efbeeac4 P 83f20e84af4c40d09a4804b6b26e697f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 07:59:11.822049 6707 raft_consensus.cc:1261] T 114b6dbafa1c4c44ac42f1d2293e3125 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Refusing update from remote peer 3b164cbbc00f4e8eb3254ddd349a3ff9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:59:11.822224 11110 tablet_copy_client.cc:308] T 6e91683ff22243c1b3877733d74e2886 P 74077b61d6f54342bf9736b4a9f2eade: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:34471
W20240501 07:59:11.822361 9125 consensus_peers.cc:485] T 2976d6353e8947b7b6c37f72efbeeac4 P 83f20e84af4c40d09a4804b6b26e697f -> Peer 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131:34471): Couldn't send request to peer 3b164cbbc00f4e8eb3254ddd349a3ff9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2976d6353e8947b7b6c37f72efbeeac4. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:11.822546 8680 consensus_queue.cc:1035] T 114b6dbafa1c4c44ac42f1d2293e3125 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:59:11.822638 6597 tablet_copy_service.cc:140] P 3b164cbbc00f4e8eb3254ddd349a3ff9: Received BeginTabletCopySession request for tablet 6e91683ff22243c1b3877733d74e2886 from peer 74077b61d6f54342bf9736b4a9f2eade ({username='slave'} at 127.0.99.133:57559)
I20240501 07:59:11.822731 6597 tablet_copy_service.cc:161] P 3b164cbbc00f4e8eb3254ddd349a3ff9: Beginning new tablet copy session on tablet 6e91683ff22243c1b3877733d74e2886 from peer 74077b61d6f54342bf9736b4a9f2eade at {username='slave'} at 127.0.99.133:57559: session id = 74077b61d6f54342bf9736b4a9f2eade-6e91683ff22243c1b3877733d74e2886
I20240501 07:59:11.822845 6444 raft_consensus.cc:2932] T 2398430a23094711a13a86cc88e27a78 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } } }
I20240501 07:59:11.823738 6597 tablet_copy_source_session.cc:205] T 6e91683ff22243c1b3877733d74e2886 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:11.823890 6445 raft_consensus.cc:1261] T 114b6dbafa1c4c44ac42f1d2293e3125 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Refusing update from remote peer 3b164cbbc00f4e8eb3254ddd349a3ff9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:59:11.824443 8680 consensus_queue.cc:1035] T 114b6dbafa1c4c44ac42f1d2293e3125 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:59:11.826220 11110 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e91683ff22243c1b3877733d74e2886. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:11.828543 11110 tablet_copy_client.cc:783] T 6e91683ff22243c1b3877733d74e2886 P 74077b61d6f54342bf9736b4a9f2eade: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:11.828943 11110 tablet_copy_client.cc:647] T 6e91683ff22243c1b3877733d74e2886 P 74077b61d6f54342bf9736b4a9f2eade: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:11.830263 8680 raft_consensus.cc:2932] T 114b6dbafa1c4c44ac42f1d2293e3125 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false } } }
I20240501 07:59:11.831326 11110 tablet_copy_client.cc:523] T 6e91683ff22243c1b3877733d74e2886 P 74077b61d6f54342bf9736b4a9f2eade: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:11.832752 11110 tablet_bootstrap.cc:492] T 6e91683ff22243c1b3877733d74e2886 P 74077b61d6f54342bf9736b4a9f2eade: Bootstrap starting.
I20240501 07:59:11.833462 9056 raft_consensus.cc:2932] T 114b6dbafa1c4c44ac42f1d2293e3125 P a505e7cb133c4392bfda8a4a20874ae9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false } } }
I20240501 07:59:11.834008 6157 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 2398430a23094711a13a86cc88e27a78 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 07:59:11.834949 6171 catalog_manager.cc:5484] T 2398430a23094711a13a86cc88e27a78 P 8b8b3ad3aa8f4053b2167dd550f2086f reported cstate change: config changed from index 5 to 6, VOTER 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 07:59:11.835413 6707 raft_consensus.cc:2932] T 114b6dbafa1c4c44ac42f1d2293e3125 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false } } }
I20240501 07:59:11.836163 10830 tablet_bootstrap.cc:492] T 782e37a4ebb34a68b461a74115ae8703 P 83f20e84af4c40d09a4804b6b26e697f: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:11.836431 10830 tablet_bootstrap.cc:492] T 782e37a4ebb34a68b461a74115ae8703 P 83f20e84af4c40d09a4804b6b26e697f: Bootstrap complete.
I20240501 07:59:11.836608 10830 ts_tablet_manager.cc:1367] T 782e37a4ebb34a68b461a74115ae8703 P 83f20e84af4c40d09a4804b6b26e697f: Time spent bootstrapping tablet: real 0.034s user 0.020s sys 0.005s
I20240501 07:59:11.837841 6165 catalog_manager.cc:5484] T 114b6dbafa1c4c44ac42f1d2293e3125 P 3b164cbbc00f4e8eb3254ddd349a3ff9 reported cstate change: config changed from index 4 to 5, a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 07:59:11.838163 10830 raft_consensus.cc:348] T 782e37a4ebb34a68b461a74115ae8703 P 83f20e84af4c40d09a4804b6b26e697f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: true } }
I20240501 07:59:11.838279 10830 raft_consensus.cc:729] T 782e37a4ebb34a68b461a74115ae8703 P 83f20e84af4c40d09a4804b6b26e697f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 83f20e84af4c40d09a4804b6b26e697f, State: Initialized, Role: LEARNER
I20240501 07:59:11.838389 10830 consensus_queue.cc:260] T 782e37a4ebb34a68b461a74115ae8703 P 83f20e84af4c40d09a4804b6b26e697f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: true } }
I20240501 07:59:11.839128 6556 tablet_service.cc:1514] Processing DeleteTablet for tablet 2398430a23094711a13a86cc88e27a78 with delete_type TABLET_DATA_TOMBSTONED (TS 3b164cbbc00f4e8eb3254ddd349a3ff9 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:53874
I20240501 07:59:11.839365 10753 tablet_replica.cc:330] T 2398430a23094711a13a86cc88e27a78 P 3b164cbbc00f4e8eb3254ddd349a3ff9: stopping tablet replica
I20240501 07:59:11.839507 10753 raft_consensus.cc:2229] T 2398430a23094711a13a86cc88e27a78 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:11.839641 10753 raft_consensus.cc:2258] T 2398430a23094711a13a86cc88e27a78 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:11.840358 10753 ts_tablet_manager.cc:1875] T 2398430a23094711a13a86cc88e27a78 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:59:11.841027 6444 raft_consensus.cc:2932] T 114b6dbafa1c4c44ac42f1d2293e3125 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false } } }
I20240501 07:59:11.844008 6313 raft_consensus.cc:1261] T 2976d6353e8947b7b6c37f72efbeeac4 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Refusing update from remote peer 83f20e84af4c40d09a4804b6b26e697f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:11.843952 8793 raft_consensus.cc:2932] T 2398430a23094711a13a86cc88e27a78 P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } } }
I20240501 07:59:11.844098 10753 ts_tablet_manager.cc:1888] T 2398430a23094711a13a86cc88e27a78 P 3b164cbbc00f4e8eb3254ddd349a3ff9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 07:59:11.844172 10753 log.cc:1196] T 2398430a23094711a13a86cc88e27a78 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/2398430a23094711a13a86cc88e27a78
I20240501 07:59:11.844704 10848 consensus_queue.cc:1035] T 2976d6353e8947b7b6c37f72efbeeac4 P 83f20e84af4c40d09a4804b6b26e697f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 07:59:11.844811 10830 ts_tablet_manager.cc:1398] T 782e37a4ebb34a68b461a74115ae8703 P 83f20e84af4c40d09a4804b6b26e697f: Time spent starting tablet: real 0.008s user 0.004s sys 0.000s
I20240501 07:59:11.845263 6466 tablet_copy_service.cc:342] P 8315d22c56f7439a9e1fc82f153f018c: Request end of tablet copy session 83f20e84af4c40d09a4804b6b26e697f-782e37a4ebb34a68b461a74115ae8703 received from {username='slave'} at 127.0.99.136:51357
I20240501 07:59:11.845350 6466 tablet_copy_service.cc:433] P 8315d22c56f7439a9e1fc82f153f018c: ending tablet copy session 83f20e84af4c40d09a4804b6b26e697f-782e37a4ebb34a68b461a74115ae8703 on tablet 782e37a4ebb34a68b461a74115ae8703 with peer 83f20e84af4c40d09a4804b6b26e697f
I20240501 07:59:11.846174 6160 catalog_manager.cc:4830] TS 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131:34471): tablet 2398430a23094711a13a86cc88e27a78 (table rebalance_test_table_19 [id=35f40c7e6b1e4d799576a7d98130d848]) successfully deleted
I20240501 07:59:11.848212 6577 consensus_queue.cc:237] T 114b6dbafa1c4c44ac42f1d2293e3125 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false } }
I20240501 07:59:11.849948 6445 raft_consensus.cc:1261] T 114b6dbafa1c4c44ac42f1d2293e3125 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Refusing update from remote peer 3b164cbbc00f4e8eb3254ddd349a3ff9: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 07:59:11.849915 10848 raft_consensus.cc:2932] T 2976d6353e8947b7b6c37f72efbeeac4 P 83f20e84af4c40d09a4804b6b26e697f [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: true } } }
I20240501 07:59:11.850540 10046 consensus_queue.cc:1035] T 114b6dbafa1c4c44ac42f1d2293e3125 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 07:59:11.850844 9056 raft_consensus.cc:1261] T 114b6dbafa1c4c44ac42f1d2293e3125 P a505e7cb133c4392bfda8a4a20874ae9 [term 1 FOLLOWER]: Refusing update from remote peer 3b164cbbc00f4e8eb3254ddd349a3ff9: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 07:59:11.851327 10046 consensus_queue.cc:1035] T 114b6dbafa1c4c44ac42f1d2293e3125 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 07:59:11.852181 9055 raft_consensus.cc:2932] T 2976d6353e8947b7b6c37f72efbeeac4 P a505e7cb133c4392bfda8a4a20874ae9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: true } } }
I20240501 07:59:11.854645 6313 raft_consensus.cc:2932] T 2976d6353e8947b7b6c37f72efbeeac4 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: true } } }
I20240501 07:59:11.855659 6171 catalog_manager.cc:5484] T 2976d6353e8947b7b6c37f72efbeeac4 P a505e7cb133c4392bfda8a4a20874ae9 reported cstate change: config changed from index -1 to 2, NON_VOTER 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131) added. New cstate: current_term: 1 leader_uuid: "83f20e84af4c40d09a4804b6b26e697f" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: true } } }
I20240501 07:59:11.860985 6165 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55950:
table { table_name: "rebalance_test_table_16" } modify_external_catalogs: true
I20240501 07:59:11.861127 6165 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55950:
table { table_name: "rebalance_test_table_16" } modify_external_catalogs: true
I20240501 07:59:11.861678 9507 raft_consensus.cc:2932] T 114b6dbafa1c4c44ac42f1d2293e3125 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false } } }
I20240501 07:59:11.864768 11110 tablet_bootstrap.cc:492] T 6e91683ff22243c1b3877733d74e2886 P 74077b61d6f54342bf9736b4a9f2eade: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:11.865216 11110 tablet_bootstrap.cc:492] T 6e91683ff22243c1b3877733d74e2886 P 74077b61d6f54342bf9736b4a9f2eade: Bootstrap complete.
I20240501 07:59:11.865201 6160 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 114b6dbafa1c4c44ac42f1d2293e3125 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 07:59:11.865494 11110 ts_tablet_manager.cc:1367] T 6e91683ff22243c1b3877733d74e2886 P 74077b61d6f54342bf9736b4a9f2eade: Time spent bootstrapping tablet: real 0.033s user 0.022s sys 0.005s
I20240501 07:59:11.865758 6171 catalog_manager.cc:5484] T 114b6dbafa1c4c44ac42f1d2293e3125 P 3b164cbbc00f4e8eb3254ddd349a3ff9 reported cstate change: config changed from index 5 to 6, VOTER 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 07:59:11.866037 11110 raft_consensus.cc:348] T 6e91683ff22243c1b3877733d74e2886 P 74077b61d6f54342bf9736b4a9f2eade [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: true } }
I20240501 07:59:11.866145 11110 raft_consensus.cc:729] T 6e91683ff22243c1b3877733d74e2886 P 74077b61d6f54342bf9736b4a9f2eade [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 74077b61d6f54342bf9736b4a9f2eade, State: Initialized, Role: LEARNER
I20240501 07:59:11.866245 11110 consensus_queue.cc:260] T 6e91683ff22243c1b3877733d74e2886 P 74077b61d6f54342bf9736b4a9f2eade [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: true } }
I20240501 07:59:11.866539 11110 ts_tablet_manager.cc:1398] T 6e91683ff22243c1b3877733d74e2886 P 74077b61d6f54342bf9736b4a9f2eade: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:59:11.867314 6597 tablet_copy_service.cc:342] P 3b164cbbc00f4e8eb3254ddd349a3ff9: Request end of tablet copy session 74077b61d6f54342bf9736b4a9f2eade-6e91683ff22243c1b3877733d74e2886 received from {username='slave'} at 127.0.99.133:57559
I20240501 07:59:11.867413 6597 tablet_copy_service.cc:433] P 3b164cbbc00f4e8eb3254ddd349a3ff9: ending tablet copy session 74077b61d6f54342bf9736b4a9f2eade-6e91683ff22243c1b3877733d74e2886 on tablet 6e91683ff22243c1b3877733d74e2886 with peer 74077b61d6f54342bf9736b4a9f2eade
I20240501 07:59:11.867897 6165 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 0bc706e85ce74cef97a4827ea4d4aa65: Sending DeleteTablet for 4 replicas of tablet 33b3834db0cd4407ae7c26e044459e87
I20240501 07:59:11.868250 6165 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 0bc706e85ce74cef97a4827ea4d4aa65: Sending DeleteTablet for 3 replicas of tablet af8baea378be41f1806ea2cae61764bb
I20240501 07:59:11.868520 6165 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 0bc706e85ce74cef97a4827ea4d4aa65: Sending DeleteTablet for 3 replicas of tablet 3b04063d87be40f7ac1b46715d457983
I20240501 07:59:11.869122 6294 tablet_service.cc:1514] Processing DeleteTablet for tablet 33b3834db0cd4407ae7c26e044459e87 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:11 UTC) from {username='slave'} at 127.0.0.1:45480
I20240501 07:59:11.869186 6425 tablet_service.cc:1514] Processing DeleteTablet for tablet 33b3834db0cd4407ae7c26e044459e87 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:11 UTC) from {username='slave'} at 127.0.0.1:45034
I20240501 07:59:11.869208 6686 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b04063d87be40f7ac1b46715d457983 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:11 UTC) from {username='slave'} at 127.0.0.1:38524
I20240501 07:59:11.869302 6687 tablet_service.cc:1514] Processing DeleteTablet for tablet af8baea378be41f1806ea2cae61764bb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:11 UTC) from {username='slave'} at 127.0.0.1:38524
I20240501 07:59:11.869340 10309 tablet_replica.cc:330] T 33b3834db0cd4407ae7c26e044459e87 P 8b8b3ad3aa8f4053b2167dd550f2086f: stopping tablet replica
I20240501 07:59:11.869405 10751 tablet_replica.cc:330] T 33b3834db0cd4407ae7c26e044459e87 P 8315d22c56f7439a9e1fc82f153f018c: stopping tablet replica
I20240501 07:59:11.869431 6424 tablet_service.cc:1514] Processing DeleteTablet for tablet af8baea378be41f1806ea2cae61764bb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:11 UTC) from {username='slave'} at 127.0.0.1:45034
I20240501 07:59:11.869503 10309 raft_consensus.cc:2229] T 33b3834db0cd4407ae7c26e044459e87 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:11.869540 10751 raft_consensus.cc:2229] T 33b3834db0cd4407ae7c26e044459e87 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:11.869565 9035 tablet_service.cc:1514] Processing DeleteTablet for tablet 33b3834db0cd4407ae7c26e044459e87 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:11 UTC) from {username='slave'} at 127.0.0.1:44236
I20240501 07:59:11.869678 10309 raft_consensus.cc:2258] T 33b3834db0cd4407ae7c26e044459e87 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:11.869701 10751 raft_consensus.cc:2258] T 33b3834db0cd4407ae7c26e044459e87 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:11.869746 10310 tablet_replica.cc:330] T 3b04063d87be40f7ac1b46715d457983 P 1f9cb72be98941b5a8e504cbe34132b9: stopping tablet replica
I20240501 07:59:11.869877 10310 raft_consensus.cc:2229] T 3b04063d87be40f7ac1b46715d457983 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:11.870007 10310 raft_consensus.cc:2258] T 3b04063d87be40f7ac1b46715d457983 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:11.869869 6444 raft_consensus.cc:2932] T 114b6dbafa1c4c44ac42f1d2293e3125 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false } } }
I20240501 07:59:11.870213 10309 ts_tablet_manager.cc:1875] T 33b3834db0cd4407ae7c26e044459e87 P 8b8b3ad3aa8f4053b2167dd550f2086f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:59:11.870249 10751 ts_tablet_manager.cc:1875] T 33b3834db0cd4407ae7c26e044459e87 P 8315d22c56f7439a9e1fc82f153f018c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:59:11.870543 10310 ts_tablet_manager.cc:1875] T 3b04063d87be40f7ac1b46715d457983 P 1f9cb72be98941b5a8e504cbe34132b9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:59:11.872422 10309 ts_tablet_manager.cc:1888] T 33b3834db0cd4407ae7c26e044459e87 P 8b8b3ad3aa8f4053b2167dd550f2086f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:59:11.872484 10309 log.cc:1196] T 33b3834db0cd4407ae7c26e044459e87 P 8b8b3ad3aa8f4053b2167dd550f2086f: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/33b3834db0cd4407ae7c26e044459e87
I20240501 07:59:11.872620 6555 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b04063d87be40f7ac1b46715d457983 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:11 UTC) from {username='slave'} at 127.0.0.1:53874
I20240501 07:59:11.872748 10309 ts_tablet_manager.cc:1909] T 33b3834db0cd4407ae7c26e044459e87 P 8b8b3ad3aa8f4053b2167dd550f2086f: Deleting consensus metadata
I20240501 07:59:11.872803 10310 ts_tablet_manager.cc:1888] T 3b04063d87be40f7ac1b46715d457983 P 1f9cb72be98941b5a8e504cbe34132b9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 07:59:11.872869 10310 log.cc:1196] T 3b04063d87be40f7ac1b46715d457983 P 1f9cb72be98941b5a8e504cbe34132b9: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/3b04063d87be40f7ac1b46715d457983
I20240501 07:59:11.873028 10753 tablet_replica.cc:330] T 3b04063d87be40f7ac1b46715d457983 P 3b164cbbc00f4e8eb3254ddd349a3ff9: stopping tablet replica
I20240501 07:59:11.873160 10310 ts_tablet_manager.cc:1909] T 3b04063d87be40f7ac1b46715d457983 P 1f9cb72be98941b5a8e504cbe34132b9: Deleting consensus metadata
I20240501 07:59:11.873203 10753 raft_consensus.cc:2229] T 3b04063d87be40f7ac1b46715d457983 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 LEADER]: Raft consensus shutting down.
I20240501 07:59:11.873396 6556 tablet_service.cc:1514] Processing DeleteTablet for tablet 33b3834db0cd4407ae7c26e044459e87 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:11 UTC) from {username='slave'} at 127.0.0.1:53874
I20240501 07:59:11.873761 10940 tablet_replica.cc:330] T 33b3834db0cd4407ae7c26e044459e87 P a505e7cb133c4392bfda8a4a20874ae9: stopping tablet replica
I20240501 07:59:11.873775 6295 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b04063d87be40f7ac1b46715d457983 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:11 UTC) from {username='slave'} at 127.0.0.1:45480
I20240501 07:59:11.872490 10751 ts_tablet_manager.cc:1888] T 33b3834db0cd4407ae7c26e044459e87 P 8315d22c56f7439a9e1fc82f153f018c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:59:11.873921 10940 raft_consensus.cc:2229] T 33b3834db0cd4407ae7c26e044459e87 P a505e7cb133c4392bfda8a4a20874ae9 [term 1 LEARNER]: Raft consensus shutting down.
I20240501 07:59:11.873988 10751 log.cc:1196] T 33b3834db0cd4407ae7c26e044459e87 P 8315d22c56f7439a9e1fc82f153f018c: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/33b3834db0cd4407ae7c26e044459e87
I20240501 07:59:11.874043 10940 raft_consensus.cc:2258] T 33b3834db0cd4407ae7c26e044459e87 P a505e7cb133c4392bfda8a4a20874ae9 [term 1 LEARNER]: Raft consensus is shut down!
I20240501 07:59:11.874271 10751 ts_tablet_manager.cc:1909] T 33b3834db0cd4407ae7c26e044459e87 P 8315d22c56f7439a9e1fc82f153f018c: Deleting consensus metadata
I20240501 07:59:11.874693 10940 ts_tablet_manager.cc:1875] T 33b3834db0cd4407ae7c26e044459e87 P a505e7cb133c4392bfda8a4a20874ae9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:59:11.874830 10751 tablet_replica.cc:330] T af8baea378be41f1806ea2cae61764bb P 8315d22c56f7439a9e1fc82f153f018c: stopping tablet replica
I20240501 07:59:11.874950 10751 raft_consensus.cc:2229] T af8baea378be41f1806ea2cae61764bb P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:11.875061 10751 raft_consensus.cc:2258] T af8baea378be41f1806ea2cae61764bb P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:11.875483 10751 ts_tablet_manager.cc:1875] T af8baea378be41f1806ea2cae61764bb P 8315d22c56f7439a9e1fc82f153f018c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:59:11.878113 6157 catalog_manager.cc:4830] TS 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129:46875): tablet 33b3834db0cd4407ae7c26e044459e87 (table rebalance_test_table_16 [id=bdae856e60144776aae709dddb837795]) successfully deleted
I20240501 07:59:11.873409 10753 raft_consensus.cc:2258] T 3b04063d87be40f7ac1b46715d457983 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:11.879590 10753 ts_tablet_manager.cc:1875] T 3b04063d87be40f7ac1b46715d457983 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:59:11.880028 6294 tablet_service.cc:1514] Processing DeleteTablet for tablet af8baea378be41f1806ea2cae61764bb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:11 UTC) from {username='slave'} at 127.0.0.1:45480
I20240501 07:59:11.880365 6159 catalog_manager.cc:4830] TS 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130:32941): tablet 33b3834db0cd4407ae7c26e044459e87 (table rebalance_test_table_16 [id=bdae856e60144776aae709dddb837795]) successfully deleted
I20240501 07:59:11.881484 9940 raft_consensus.cc:482] T 12ce6a21d06c4ba889542758be9dc62d P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:11.881574 9940 raft_consensus.cc:504] T 12ce6a21d06c4ba889542758be9dc62d P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } }
I20240501 07:59:11.881949 9940 leader_election.cc:288] T 12ce6a21d06c4ba889542758be9dc62d P a505e7cb133c4392bfda8a4a20874ae9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134:40141), 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133:39859)
I20240501 07:59:11.882416 8924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12ce6a21d06c4ba889542758be9dc62d" candidate_uuid: "a505e7cb133c4392bfda8a4a20874ae9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13d05ebdca8c4f489c597ce550dd773d" is_pre_election: true
I20240501 07:59:11.882565 8924 raft_consensus.cc:2454] T 12ce6a21d06c4ba889542758be9dc62d P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a505e7cb133c4392bfda8a4a20874ae9 in term 0.
I20240501 07:59:11.882639 8793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12ce6a21d06c4ba889542758be9dc62d" candidate_uuid: "a505e7cb133c4392bfda8a4a20874ae9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74077b61d6f54342bf9736b4a9f2eade" is_pre_election: true
I20240501 07:59:11.882771 8793 raft_consensus.cc:2454] T 12ce6a21d06c4ba889542758be9dc62d P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a505e7cb133c4392bfda8a4a20874ae9 in term 0.
I20240501 07:59:11.882880 8991 leader_election.cc:302] T 12ce6a21d06c4ba889542758be9dc62d P a505e7cb133c4392bfda8a4a20874ae9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13d05ebdca8c4f489c597ce550dd773d, a505e7cb133c4392bfda8a4a20874ae9; no voters:
I20240501 07:59:11.884678 6687 tablet_service.cc:1514] Processing DeleteTablet for tablet 114b6dbafa1c4c44ac42f1d2293e3125 with delete_type TABLET_DATA_TOMBSTONED (TS 1f9cb72be98941b5a8e504cbe34132b9 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:38524
I20240501 07:59:11.884995 9940 raft_consensus.cc:2781] T 12ce6a21d06c4ba889542758be9dc62d P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:11.885066 9940 raft_consensus.cc:482] T 12ce6a21d06c4ba889542758be9dc62d P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:11.885116 9940 raft_consensus.cc:3037] T 12ce6a21d06c4ba889542758be9dc62d P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:11.885308 10753 ts_tablet_manager.cc:1888] T 3b04063d87be40f7ac1b46715d457983 P 3b164cbbc00f4e8eb3254ddd349a3ff9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 07:59:11.885371 10753 log.cc:1196] T 3b04063d87be40f7ac1b46715d457983 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/3b04063d87be40f7ac1b46715d457983
I20240501 07:59:11.885627 10753 ts_tablet_manager.cc:1909] T 3b04063d87be40f7ac1b46715d457983 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Deleting consensus metadata
I20240501 07:59:11.886157 9940 raft_consensus.cc:504] T 12ce6a21d06c4ba889542758be9dc62d P a505e7cb133c4392bfda8a4a20874ae9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } }
I20240501 07:59:11.886569 9940 leader_election.cc:288] T 12ce6a21d06c4ba889542758be9dc62d P a505e7cb133c4392bfda8a4a20874ae9 [CANDIDATE]: Term 1 election: Requested vote from peers 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134:40141), 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133:39859)
I20240501 07:59:11.887008 8793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12ce6a21d06c4ba889542758be9dc62d" candidate_uuid: "a505e7cb133c4392bfda8a4a20874ae9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74077b61d6f54342bf9736b4a9f2eade"
I20240501 07:59:11.887066 10309 tablet_replica.cc:330] T 3b04063d87be40f7ac1b46715d457983 P 8b8b3ad3aa8f4053b2167dd550f2086f: stopping tablet replica
I20240501 07:59:11.887128 8793 raft_consensus.cc:3037] T 12ce6a21d06c4ba889542758be9dc62d P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:11.887295 10309 raft_consensus.cc:2229] T 3b04063d87be40f7ac1b46715d457983 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:11.887439 10309 raft_consensus.cc:2258] T 3b04063d87be40f7ac1b46715d457983 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:11.887959 10309 ts_tablet_manager.cc:1875] T 3b04063d87be40f7ac1b46715d457983 P 8b8b3ad3aa8f4053b2167dd550f2086f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:59:11.888087 8793 raft_consensus.cc:2454] T 12ce6a21d06c4ba889542758be9dc62d P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a505e7cb133c4392bfda8a4a20874ae9 in term 1.
I20240501 07:59:11.888428 8989 leader_election.cc:302] T 12ce6a21d06c4ba889542758be9dc62d P a505e7cb133c4392bfda8a4a20874ae9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74077b61d6f54342bf9736b4a9f2eade, a505e7cb133c4392bfda8a4a20874ae9; no voters:
I20240501 07:59:11.889001 10753 tablet_replica.cc:330] T 33b3834db0cd4407ae7c26e044459e87 P 3b164cbbc00f4e8eb3254ddd349a3ff9: stopping tablet replica
I20240501 07:59:11.889005 8924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12ce6a21d06c4ba889542758be9dc62d" candidate_uuid: "a505e7cb133c4392bfda8a4a20874ae9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13d05ebdca8c4f489c597ce550dd773d"
I20240501 07:59:11.889138 8924 raft_consensus.cc:3037] T 12ce6a21d06c4ba889542758be9dc62d P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:11.889163 6160 catalog_manager.cc:4830] TS 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131:34471): tablet 3b04063d87be40f7ac1b46715d457983 (table rebalance_test_table_16 [id=bdae856e60144776aae709dddb837795]) successfully deleted
I20240501 07:59:11.889205 10753 raft_consensus.cc:2229] T 33b3834db0cd4407ae7c26e044459e87 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 LEADER]: Raft consensus shutting down.
I20240501 07:59:11.889474 10753 raft_consensus.cc:2258] T 33b3834db0cd4407ae7c26e044459e87 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:11.890007 10753 ts_tablet_manager.cc:1875] T 33b3834db0cd4407ae7c26e044459e87 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:59:11.890174 8924 raft_consensus.cc:2454] T 12ce6a21d06c4ba889542758be9dc62d P 13d05ebdca8c4f489c597ce550dd773d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a505e7cb133c4392bfda8a4a20874ae9 in term 1.
I20240501 07:59:11.891307 9940 raft_consensus.cc:2781] T 12ce6a21d06c4ba889542758be9dc62d P a505e7cb133c4392bfda8a4a20874ae9 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:11.891435 9940 raft_consensus.cc:686] T 12ce6a21d06c4ba889542758be9dc62d P a505e7cb133c4392bfda8a4a20874ae9 [term 1 LEADER]: Becoming Leader. State: Replica: a505e7cb133c4392bfda8a4a20874ae9, State: Running, Role: LEADER
I20240501 07:59:11.891322 9056 raft_consensus.cc:2932] T 114b6dbafa1c4c44ac42f1d2293e3125 P a505e7cb133c4392bfda8a4a20874ae9 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false } } }
I20240501 07:59:11.891564 9940 consensus_queue.cc:237] T 12ce6a21d06c4ba889542758be9dc62d P a505e7cb133c4392bfda8a4a20874ae9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } }
I20240501 07:59:11.893684 10940 ts_tablet_manager.cc:1888] T 33b3834db0cd4407ae7c26e044459e87 P a505e7cb133c4392bfda8a4a20874ae9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:59:11.893769 10940 log.cc:1196] T 33b3834db0cd4407ae7c26e044459e87 P a505e7cb133c4392bfda8a4a20874ae9: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/33b3834db0cd4407ae7c26e044459e87
I20240501 07:59:11.894078 10940 ts_tablet_manager.cc:1909] T 33b3834db0cd4407ae7c26e044459e87 P a505e7cb133c4392bfda8a4a20874ae9: Deleting consensus metadata
I20240501 07:59:11.894771 6158 catalog_manager.cc:4830] TS a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): tablet 33b3834db0cd4407ae7c26e044459e87 (table rebalance_test_table_16 [id=bdae856e60144776aae709dddb837795]) successfully deleted
I20240501 07:59:11.894529 6168 catalog_manager.cc:5484] T 12ce6a21d06c4ba889542758be9dc62d P a505e7cb133c4392bfda8a4a20874ae9 reported cstate change: term changed from 0 to 1, leader changed from <none> to a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135). New cstate: current_term: 1 leader_uuid: "a505e7cb133c4392bfda8a4a20874ae9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } health_report { overall_health: UNKNOWN } } }
I20240501 07:59:11.895589 6159 catalog_manager.cc:4830] TS 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132:36773): tablet 3b04063d87be40f7ac1b46715d457983 (table rebalance_test_table_16 [id=bdae856e60144776aae709dddb837795]) successfully deleted
I20240501 07:59:11.895835 10310 tablet_replica.cc:330] T af8baea378be41f1806ea2cae61764bb P 1f9cb72be98941b5a8e504cbe34132b9: stopping tablet replica
I20240501 07:59:11.896068 10310 raft_consensus.cc:2229] T af8baea378be41f1806ea2cae61764bb P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 LEADER]: Raft consensus shutting down.
I20240501 07:59:11.896318 10310 raft_consensus.cc:2258] T af8baea378be41f1806ea2cae61764bb P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:11.897011 10310 ts_tablet_manager.cc:1875] T af8baea378be41f1806ea2cae61764bb P 1f9cb72be98941b5a8e504cbe34132b9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:59:11.898970 10751 ts_tablet_manager.cc:1888] T af8baea378be41f1806ea2cae61764bb P 8315d22c56f7439a9e1fc82f153f018c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 07:59:11.899796 10309 ts_tablet_manager.cc:1888] T 3b04063d87be40f7ac1b46715d457983 P 8b8b3ad3aa8f4053b2167dd550f2086f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 07:59:11.900784 10753 ts_tablet_manager.cc:1888] T 33b3834db0cd4407ae7c26e044459e87 P 3b164cbbc00f4e8eb3254ddd349a3ff9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:59:11.900784 10310 ts_tablet_manager.cc:1888] T af8baea378be41f1806ea2cae61764bb P 1f9cb72be98941b5a8e504cbe34132b9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 07:59:11.900861 10310 log.cc:1196] T af8baea378be41f1806ea2cae61764bb P 1f9cb72be98941b5a8e504cbe34132b9: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/af8baea378be41f1806ea2cae61764bb
I20240501 07:59:11.899066 10751 log.cc:1196] T af8baea378be41f1806ea2cae61764bb P 8315d22c56f7439a9e1fc82f153f018c: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/af8baea378be41f1806ea2cae61764bb
I20240501 07:59:11.901444 10751 ts_tablet_manager.cc:1909] T af8baea378be41f1806ea2cae61764bb P 8315d22c56f7439a9e1fc82f153f018c: Deleting consensus metadata
I20240501 07:59:11.901567 10309 log.cc:1196] T 3b04063d87be40f7ac1b46715d457983 P 8b8b3ad3aa8f4053b2167dd550f2086f: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/3b04063d87be40f7ac1b46715d457983
I20240501 07:59:11.901932 10309 ts_tablet_manager.cc:1909] T 3b04063d87be40f7ac1b46715d457983 P 8b8b3ad3aa8f4053b2167dd550f2086f: Deleting consensus metadata
I20240501 07:59:11.902570 10310 ts_tablet_manager.cc:1909] T af8baea378be41f1806ea2cae61764bb P 1f9cb72be98941b5a8e504cbe34132b9: Deleting consensus metadata
I20240501 07:59:11.902689 10753 log.cc:1196] T 33b3834db0cd4407ae7c26e044459e87 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/33b3834db0cd4407ae7c26e044459e87
I20240501 07:59:11.903242 10753 ts_tablet_manager.cc:1909] T 33b3834db0cd4407ae7c26e044459e87 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Deleting consensus metadata
I20240501 07:59:11.903285 6159 catalog_manager.cc:4830] TS 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132:36773): tablet af8baea378be41f1806ea2cae61764bb (table rebalance_test_table_16 [id=bdae856e60144776aae709dddb837795]) successfully deleted
I20240501 07:59:11.903566 10310 tablet_replica.cc:330] T 114b6dbafa1c4c44ac42f1d2293e3125 P 1f9cb72be98941b5a8e504cbe34132b9: stopping tablet replica
I20240501 07:59:11.903708 10310 raft_consensus.cc:2229] T 114b6dbafa1c4c44ac42f1d2293e3125 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:11.903889 10310 raft_consensus.cc:2258] T 114b6dbafa1c4c44ac42f1d2293e3125 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:11.904191 6159 catalog_manager.cc:4830] TS 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130:32941): tablet af8baea378be41f1806ea2cae61764bb (table rebalance_test_table_16 [id=bdae856e60144776aae709dddb837795]) successfully deleted
I20240501 07:59:11.904733 10310 ts_tablet_manager.cc:1875] T 114b6dbafa1c4c44ac42f1d2293e3125 P 1f9cb72be98941b5a8e504cbe34132b9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:59:11.904860 6157 catalog_manager.cc:4830] TS 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129:46875): tablet 3b04063d87be40f7ac1b46715d457983 (table rebalance_test_table_16 [id=bdae856e60144776aae709dddb837795]) successfully deleted
I20240501 07:59:11.906060 10309 tablet_replica.cc:330] T af8baea378be41f1806ea2cae61764bb P 8b8b3ad3aa8f4053b2167dd550f2086f: stopping tablet replica
I20240501 07:59:11.906656 10310 ts_tablet_manager.cc:1888] T 114b6dbafa1c4c44ac42f1d2293e3125 P 1f9cb72be98941b5a8e504cbe34132b9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 07:59:11.906723 10310 log.cc:1196] T 114b6dbafa1c4c44ac42f1d2293e3125 P 1f9cb72be98941b5a8e504cbe34132b9: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/114b6dbafa1c4c44ac42f1d2293e3125
I20240501 07:59:11.907186 6159 catalog_manager.cc:4830] TS 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132:36773): tablet 114b6dbafa1c4c44ac42f1d2293e3125 (table rebalance_test_table_5 [id=336b9f044a794b6a8a51d76e720b0ecd]) successfully deleted
I20240501 07:59:11.907527 10309 raft_consensus.cc:2229] T af8baea378be41f1806ea2cae61764bb P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:11.907685 6160 catalog_manager.cc:4830] TS 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131:34471): tablet 33b3834db0cd4407ae7c26e044459e87 (table rebalance_test_table_16 [id=bdae856e60144776aae709dddb837795]) successfully deleted
I20240501 07:59:11.908097 10309 raft_consensus.cc:2258] T af8baea378be41f1806ea2cae61764bb P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:11.909224 10309 ts_tablet_manager.cc:1875] T af8baea378be41f1806ea2cae61764bb P 8b8b3ad3aa8f4053b2167dd550f2086f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:59:11.910859 11095 raft_consensus.cc:1050] T 6292ff0c7e624dd7b24715b83a30fff8 P 74077b61d6f54342bf9736b4a9f2eade: attempting to promote NON_VOTER 1f9cb72be98941b5a8e504cbe34132b9 to VOTER
I20240501 07:59:11.911276 11095 consensus_queue.cc:237] T 6292ff0c7e624dd7b24715b83a30fff8 P 74077b61d6f54342bf9736b4a9f2eade [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { replace: true } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { promote: false } }
I20240501 07:59:11.912492 6707 raft_consensus.cc:1261] T 6292ff0c7e624dd7b24715b83a30fff8 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 LEARNER]: Refusing update from remote peer 74077b61d6f54342bf9736b4a9f2eade: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 07:59:11.912693 6577 raft_consensus.cc:1261] T 6292ff0c7e624dd7b24715b83a30fff8 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Refusing update from remote peer 74077b61d6f54342bf9736b4a9f2eade: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 07:59:11.912704 9189 raft_consensus.cc:1261] T 6292ff0c7e624dd7b24715b83a30fff8 P 83f20e84af4c40d09a4804b6b26e697f [term 1 FOLLOWER]: Refusing update from remote peer 74077b61d6f54342bf9736b4a9f2eade: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 07:59:11.912833 10069 consensus_queue.cc:1035] T 6292ff0c7e624dd7b24715b83a30fff8 P 74077b61d6f54342bf9736b4a9f2eade [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 07:59:11.913133 10528 consensus_queue.cc:1035] T 6292ff0c7e624dd7b24715b83a30fff8 P 74077b61d6f54342bf9736b4a9f2eade [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 07:59:11.913475 10069 consensus_queue.cc:1035] T 6292ff0c7e624dd7b24715b83a30fff8 P 74077b61d6f54342bf9736b4a9f2eade [LEADER]: Connected to new peer: Peer: permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 07:59:11.917126 10309 ts_tablet_manager.cc:1888] T af8baea378be41f1806ea2cae61764bb P 8b8b3ad3aa8f4053b2167dd550f2086f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 07:59:11.917228 10309 log.cc:1196] T af8baea378be41f1806ea2cae61764bb P 8b8b3ad3aa8f4053b2167dd550f2086f: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/af8baea378be41f1806ea2cae61764bb
I20240501 07:59:11.917526 10309 ts_tablet_manager.cc:1909] T af8baea378be41f1806ea2cae61764bb P 8b8b3ad3aa8f4053b2167dd550f2086f: Deleting consensus metadata
I20240501 07:59:11.918280 10403 raft_consensus.cc:2932] T 6292ff0c7e624dd7b24715b83a30fff8 P 74077b61d6f54342bf9736b4a9f2eade [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { replace: true } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { promote: false } } }
I20240501 07:59:11.918474 6157 catalog_manager.cc:4830] TS 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129:46875): tablet af8baea378be41f1806ea2cae61764bb (table rebalance_test_table_16 [id=bdae856e60144776aae709dddb837795]) successfully deleted
I20240501 07:59:11.918808 9189 raft_consensus.cc:2932] T 6292ff0c7e624dd7b24715b83a30fff8 P 83f20e84af4c40d09a4804b6b26e697f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { replace: true } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { promote: false } } }
I20240501 07:59:11.921206 6707 raft_consensus.cc:2932] T 6292ff0c7e624dd7b24715b83a30fff8 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { replace: true } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { promote: false } } }
I20240501 07:59:11.921870 6168 catalog_manager.cc:5484] T 6292ff0c7e624dd7b24715b83a30fff8 P 74077b61d6f54342bf9736b4a9f2eade reported cstate change: config changed from index 2 to 3, 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "74077b61d6f54342bf9736b4a9f2eade" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 07:59:11.922466 6577 raft_consensus.cc:2932] T 6292ff0c7e624dd7b24715b83a30fff8 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { replace: true } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { promote: false } } }
I20240501 07:59:11.927513 6168 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55950:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 07:59:11.928061 8793 consensus_queue.cc:237] T 6292ff0c7e624dd7b24715b83a30fff8 P 74077b61d6f54342bf9736b4a9f2eade [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { promote: false } }
I20240501 07:59:11.929270 6707 raft_consensus.cc:1261] T 6292ff0c7e624dd7b24715b83a30fff8 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Refusing update from remote peer 74077b61d6f54342bf9736b4a9f2eade: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:59:11.929589 6577 raft_consensus.cc:1261] T 6292ff0c7e624dd7b24715b83a30fff8 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Refusing update from remote peer 74077b61d6f54342bf9736b4a9f2eade: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:59:11.929708 10528 consensus_queue.cc:1035] T 6292ff0c7e624dd7b24715b83a30fff8 P 74077b61d6f54342bf9736b4a9f2eade [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:11.929986 11095 consensus_queue.cc:1035] T 6292ff0c7e624dd7b24715b83a30fff8 P 74077b61d6f54342bf9736b4a9f2eade [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:11.931429 11095 raft_consensus.cc:2932] T 6292ff0c7e624dd7b24715b83a30fff8 P 74077b61d6f54342bf9736b4a9f2eade [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { promote: false } } }
I20240501 07:59:11.931947 8793 raft_consensus.cc:1261] T 1eae52e7f97f45b48697c624192aa14b P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Refusing update from remote peer 83f20e84af4c40d09a4804b6b26e697f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:11.932191 6707 raft_consensus.cc:1261] T 1eae52e7f97f45b48697c624192aa14b P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Refusing update from remote peer 83f20e84af4c40d09a4804b6b26e697f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:11.932325 11109 consensus_queue.cc:1035] T 1eae52e7f97f45b48697c624192aa14b P 83f20e84af4c40d09a4804b6b26e697f [LEADER]: Connected to new peer: Peer: permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:11.932570 11122 consensus_queue.cc:1035] T 1eae52e7f97f45b48697c624192aa14b P 83f20e84af4c40d09a4804b6b26e697f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:11.932870 6577 raft_consensus.cc:2932] T 6292ff0c7e624dd7b24715b83a30fff8 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { promote: false } } }
I20240501 07:59:11.934394 6157 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6292ff0c7e624dd7b24715b83a30fff8 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 07:59:11.935388 6168 catalog_manager.cc:5484] T 6292ff0c7e624dd7b24715b83a30fff8 P 74077b61d6f54342bf9736b4a9f2eade reported cstate change: config changed from index 3 to 4, VOTER 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136) evicted. New cstate: current_term: 1 leader_uuid: "74077b61d6f54342bf9736b4a9f2eade" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 07:59:11.935432 6708 raft_consensus.cc:2932] T 6292ff0c7e624dd7b24715b83a30fff8 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { promote: false } } }
I20240501 07:59:11.935684 6168 catalog_manager.cc:5576] TS 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133:39859) does not have the latest schema for tablet 1eae52e7f97f45b48697c624192aa14b (table rebalancer_dynamic_table [id=a57717a5df61497abe35bc282984a9ab]). Expected version 1 got 0
I20240501 07:59:11.939700 6164 catalog_manager.cc:5576] TS 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149) does not have the latest schema for tablet 1eae52e7f97f45b48697c624192aa14b (table rebalancer_dynamic_table [id=a57717a5df61497abe35bc282984a9ab]). Expected version 1 got 0
I20240501 07:59:11.940734 11148 tablet.cc:1722] T 1eae52e7f97f45b48697c624192aa14b P 1f9cb72be98941b5a8e504cbe34132b9: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 07:59:11.941287 9167 tablet_service.cc:1514] Processing DeleteTablet for tablet 6292ff0c7e624dd7b24715b83a30fff8 with delete_type TABLET_DATA_TOMBSTONED (TS 83f20e84af4c40d09a4804b6b26e697f not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:33594
I20240501 07:59:11.941763 11148 tablet.cc:1620] T 1eae52e7f97f45b48697c624192aa14b P 1f9cb72be98941b5a8e504cbe34132b9: MemRowSet was empty: no flush needed.
I20240501 07:59:11.941896 6171 catalog_manager.cc:5576] TS 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132:36773) does not have the latest schema for tablet 1eae52e7f97f45b48697c624192aa14b (table rebalancer_dynamic_table [id=a57717a5df61497abe35bc282984a9ab]). Expected version 1 got 0
I20240501 07:59:11.938113 11146 tablet.cc:1722] T 1eae52e7f97f45b48697c624192aa14b P 83f20e84af4c40d09a4804b6b26e697f: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 07:59:11.942896 11146 tablet.cc:1620] T 1eae52e7f97f45b48697c624192aa14b P 83f20e84af4c40d09a4804b6b26e697f: MemRowSet was empty: no flush needed.
I20240501 07:59:11.943164 6171 catalog_manager.cc:5576] TS 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149) does not have the latest schema for tablet 1eae52e7f97f45b48697c624192aa14b (table rebalancer_dynamic_table [id=a57717a5df61497abe35bc282984a9ab]). Expected version 1 got 0
I20240501 07:59:11.944054 10111 raft_consensus.cc:1050] T e92d6aeeb3df4ea3852c6ebbed0e07c5 P 8315d22c56f7439a9e1fc82f153f018c: attempting to promote NON_VOTER 13d05ebdca8c4f489c597ce550dd773d to VOTER
I20240501 07:59:11.944458 10111 consensus_queue.cc:237] T e92d6aeeb3df4ea3852c6ebbed0e07c5 P 8315d22c56f7439a9e1fc82f153f018c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false } }
I20240501 07:59:11.946197 6168 catalog_manager.cc:5576] TS 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133:39859) does not have the latest schema for tablet 1eae52e7f97f45b48697c624192aa14b (table rebalancer_dynamic_table [id=a57717a5df61497abe35bc282984a9ab]). Expected version 1 got 0
I20240501 07:59:11.946379 8924 raft_consensus.cc:1261] T e92d6aeeb3df4ea3852c6ebbed0e07c5 P 13d05ebdca8c4f489c597ce550dd773d [term 1 LEARNER]: Refusing update from remote peer 8315d22c56f7439a9e1fc82f153f018c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:59:11.946659 6313 raft_consensus.cc:1261] T e92d6aeeb3df4ea3852c6ebbed0e07c5 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Refusing update from remote peer 8315d22c56f7439a9e1fc82f153f018c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:59:11.946820 6172 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 0bc706e85ce74cef97a4827ea4d4aa65: rebalancer_dynamic_table [id=a57717a5df61497abe35bc282984a9ab] alter complete (version 1)
I20240501 07:59:11.946792 9116 consensus_queue.cc:1035] T e92d6aeeb3df4ea3852c6ebbed0e07c5 P 8315d22c56f7439a9e1fc82f153f018c [LEADER]: Connected to new peer: Peer: permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:59:11.946841 6577 raft_consensus.cc:1261] T e92d6aeeb3df4ea3852c6ebbed0e07c5 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Refusing update from remote peer 8315d22c56f7439a9e1fc82f153f018c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:59:11.947245 9116 consensus_queue.cc:1035] T e92d6aeeb3df4ea3852c6ebbed0e07c5 P 8315d22c56f7439a9e1fc82f153f018c [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:59:11.947376 8512 consensus_queue.cc:1035] T e92d6aeeb3df4ea3852c6ebbed0e07c5 P 8315d22c56f7439a9e1fc82f153f018c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:59:11.950470 9116 raft_consensus.cc:2932] T e92d6aeeb3df4ea3852c6ebbed0e07c5 P 8315d22c56f7439a9e1fc82f153f018c [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false } } }
I20240501 07:59:11.951160 11153 tablet_replica.cc:330] T 6292ff0c7e624dd7b24715b83a30fff8 P 83f20e84af4c40d09a4804b6b26e697f: stopping tablet replica
I20240501 07:59:11.951285 11153 raft_consensus.cc:2229] T 6292ff0c7e624dd7b24715b83a30fff8 P 83f20e84af4c40d09a4804b6b26e697f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:11.951445 11153 raft_consensus.cc:2258] T 6292ff0c7e624dd7b24715b83a30fff8 P 83f20e84af4c40d09a4804b6b26e697f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:11.952522 6313 raft_consensus.cc:2932] T e92d6aeeb3df4ea3852c6ebbed0e07c5 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false } } }
I20240501 07:59:11.952682 11153 ts_tablet_manager.cc:1875] T 6292ff0c7e624dd7b24715b83a30fff8 P 83f20e84af4c40d09a4804b6b26e697f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:59:11.953049 8924 raft_consensus.cc:2932] T e92d6aeeb3df4ea3852c6ebbed0e07c5 P 13d05ebdca8c4f489c597ce550dd773d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false } } }
I20240501 07:59:11.956122 6171 catalog_manager.cc:5484] T e92d6aeeb3df4ea3852c6ebbed0e07c5 P 8b8b3ad3aa8f4053b2167dd550f2086f reported cstate change: config changed from index 4 to 5, 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8315d22c56f7439a9e1fc82f153f018c" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false } } }
I20240501 07:59:11.956292 11153 ts_tablet_manager.cc:1888] T 6292ff0c7e624dd7b24715b83a30fff8 P 83f20e84af4c40d09a4804b6b26e697f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 07:59:11.956362 11153 log.cc:1196] T 6292ff0c7e624dd7b24715b83a30fff8 P 83f20e84af4c40d09a4804b6b26e697f: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/6292ff0c7e624dd7b24715b83a30fff8
I20240501 07:59:11.957036 11147 tablet.cc:1722] T 1eae52e7f97f45b48697c624192aa14b P 74077b61d6f54342bf9736b4a9f2eade: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 07:59:11.957234 6158 catalog_manager.cc:4830] TS 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): tablet 6292ff0c7e624dd7b24715b83a30fff8 (table rebalancer_extra_table_0 [id=9ec2540a0f4b49ada014e0c68ec3b2a0]) successfully deleted
I20240501 07:59:11.957815 11147 tablet.cc:1620] T 1eae52e7f97f45b48697c624192aa14b P 74077b61d6f54342bf9736b4a9f2eade: MemRowSet was empty: no flush needed.
I20240501 07:59:11.958482 6577 raft_consensus.cc:2932] T e92d6aeeb3df4ea3852c6ebbed0e07c5 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false } } }
I20240501 07:59:11.964484 6444 consensus_queue.cc:237] T e92d6aeeb3df4ea3852c6ebbed0e07c5 P 8315d22c56f7439a9e1fc82f153f018c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false } }
I20240501 07:59:11.965715 8924 raft_consensus.cc:1261] T e92d6aeeb3df4ea3852c6ebbed0e07c5 P 13d05ebdca8c4f489c597ce550dd773d [term 1 FOLLOWER]: Refusing update from remote peer 8315d22c56f7439a9e1fc82f153f018c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 07:59:11.965701 6313 raft_consensus.cc:1261] T e92d6aeeb3df4ea3852c6ebbed0e07c5 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Refusing update from remote peer 8315d22c56f7439a9e1fc82f153f018c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 07:59:11.966105 9116 consensus_queue.cc:1035] T e92d6aeeb3df4ea3852c6ebbed0e07c5 P 8315d22c56f7439a9e1fc82f153f018c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 07:59:11.966302 10235 raft_consensus.cc:482] T 86e28c076f6749ceafe612bf730bfa68 P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:11.966398 10235 raft_consensus.cc:504] T 86e28c076f6749ceafe612bf730bfa68 P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } }
I20240501 07:59:11.966475 9116 consensus_queue.cc:1035] T e92d6aeeb3df4ea3852c6ebbed0e07c5 P 8315d22c56f7439a9e1fc82f153f018c [LEADER]: Connected to new peer: Peer: permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 07:59:11.966801 10235 leader_election.cc:288] T 86e28c076f6749ceafe612bf730bfa68 P 13d05ebdca8c4f489c597ce550dd773d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133:39859), 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129:46875)
I20240501 07:59:11.967357 8793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86e28c076f6749ceafe612bf730bfa68" candidate_uuid: "13d05ebdca8c4f489c597ce550dd773d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74077b61d6f54342bf9736b4a9f2eade" is_pre_election: true
I20240501 07:59:11.967491 8793 raft_consensus.cc:2454] T 86e28c076f6749ceafe612bf730bfa68 P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 13d05ebdca8c4f489c597ce550dd773d in term 0.
I20240501 07:59:11.967873 6311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86e28c076f6749ceafe612bf730bfa68" candidate_uuid: "13d05ebdca8c4f489c597ce550dd773d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" is_pre_election: true
I20240501 07:59:11.967984 6311 raft_consensus.cc:2454] T 86e28c076f6749ceafe612bf730bfa68 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 13d05ebdca8c4f489c597ce550dd773d in term 0.
I20240501 07:59:11.968312 8858 leader_election.cc:302] T 86e28c076f6749ceafe612bf730bfa68 P 13d05ebdca8c4f489c597ce550dd773d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13d05ebdca8c4f489c597ce550dd773d, 74077b61d6f54342bf9736b4a9f2eade; no voters:
I20240501 07:59:11.968637 10235 raft_consensus.cc:2781] T 86e28c076f6749ceafe612bf730bfa68 P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:11.968518 8512 raft_consensus.cc:2932] T e92d6aeeb3df4ea3852c6ebbed0e07c5 P 8315d22c56f7439a9e1fc82f153f018c [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false } } }
I20240501 07:59:11.968695 10235 raft_consensus.cc:482] T 86e28c076f6749ceafe612bf730bfa68 P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:11.968732 10235 raft_consensus.cc:3037] T 86e28c076f6749ceafe612bf730bfa68 P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:11.968865 6311 raft_consensus.cc:2932] T e92d6aeeb3df4ea3852c6ebbed0e07c5 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false } } }
I20240501 07:59:11.969588 10235 raft_consensus.cc:504] T 86e28c076f6749ceafe612bf730bfa68 P 13d05ebdca8c4f489c597ce550dd773d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } }
I20240501 07:59:11.969877 10235 leader_election.cc:288] T 86e28c076f6749ceafe612bf730bfa68 P 13d05ebdca8c4f489c597ce550dd773d [CANDIDATE]: Term 1 election: Requested vote from peers 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133:39859), 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129:46875)
I20240501 07:59:11.970237 6159 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e92d6aeeb3df4ea3852c6ebbed0e07c5 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 07:59:11.970394 8793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86e28c076f6749ceafe612bf730bfa68" candidate_uuid: "13d05ebdca8c4f489c597ce550dd773d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74077b61d6f54342bf9736b4a9f2eade"
I20240501 07:59:11.970515 8793 raft_consensus.cc:3037] T 86e28c076f6749ceafe612bf730bfa68 P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:11.970803 8924 raft_consensus.cc:2932] T e92d6aeeb3df4ea3852c6ebbed0e07c5 P 13d05ebdca8c4f489c597ce550dd773d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false } } }
I20240501 07:59:11.971325 8793 raft_consensus.cc:2454] T 86e28c076f6749ceafe612bf730bfa68 P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13d05ebdca8c4f489c597ce550dd773d in term 1.
I20240501 07:59:11.971779 8858 leader_election.cc:302] T 86e28c076f6749ceafe612bf730bfa68 P 13d05ebdca8c4f489c597ce550dd773d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13d05ebdca8c4f489c597ce550dd773d, 74077b61d6f54342bf9736b4a9f2eade; no voters:
I20240501 07:59:11.971980 10235 raft_consensus.cc:2781] T 86e28c076f6749ceafe612bf730bfa68 P 13d05ebdca8c4f489c597ce550dd773d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:11.972055 10235 raft_consensus.cc:686] T 86e28c076f6749ceafe612bf730bfa68 P 13d05ebdca8c4f489c597ce550dd773d [term 1 LEADER]: Becoming Leader. State: Replica: 13d05ebdca8c4f489c597ce550dd773d, State: Running, Role: LEADER
I20240501 07:59:11.972183 6311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86e28c076f6749ceafe612bf730bfa68" candidate_uuid: "13d05ebdca8c4f489c597ce550dd773d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f"
I20240501 07:59:11.972195 10235 consensus_queue.cc:237] T 86e28c076f6749ceafe612bf730bfa68 P 13d05ebdca8c4f489c597ce550dd773d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } }
I20240501 07:59:11.972278 6311 raft_consensus.cc:3037] T 86e28c076f6749ceafe612bf730bfa68 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:11.972203 6171 catalog_manager.cc:5484] T e92d6aeeb3df4ea3852c6ebbed0e07c5 P 8315d22c56f7439a9e1fc82f153f018c reported cstate change: config changed from index 5 to 6, VOTER 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "8315d22c56f7439a9e1fc82f153f018c" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 07:59:11.973250 6311 raft_consensus.cc:2454] T 86e28c076f6749ceafe612bf730bfa68 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13d05ebdca8c4f489c597ce550dd773d in term 1.
I20240501 07:59:11.975780 6556 tablet_service.cc:1514] Processing DeleteTablet for tablet e92d6aeeb3df4ea3852c6ebbed0e07c5 with delete_type TABLET_DATA_TOMBSTONED (TS 3b164cbbc00f4e8eb3254ddd349a3ff9 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:53874
I20240501 07:59:11.975996 6168 catalog_manager.cc:5484] T 86e28c076f6749ceafe612bf730bfa68 P 13d05ebdca8c4f489c597ce550dd773d reported cstate change: term changed from 0 to 1, leader changed from <none> to 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134). New cstate: current_term: 1 leader_uuid: "13d05ebdca8c4f489c597ce550dd773d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } health_report { overall_health: UNKNOWN } } }
I20240501 07:59:11.976279 10753 tablet_replica.cc:330] T e92d6aeeb3df4ea3852c6ebbed0e07c5 P 3b164cbbc00f4e8eb3254ddd349a3ff9: stopping tablet replica
I20240501 07:59:11.976428 10753 raft_consensus.cc:2229] T e92d6aeeb3df4ea3852c6ebbed0e07c5 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:11.976605 10753 raft_consensus.cc:2258] T e92d6aeeb3df4ea3852c6ebbed0e07c5 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:11.977505 10753 ts_tablet_manager.cc:1875] T e92d6aeeb3df4ea3852c6ebbed0e07c5 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:59:11.977686 9618 raft_consensus.cc:1050] T b284ed2cb0264ced99a4ffd90983d902 P 1f9cb72be98941b5a8e504cbe34132b9: attempting to promote NON_VOTER 13d05ebdca8c4f489c597ce550dd773d to VOTER
I20240501 07:59:11.978184 9618 consensus_queue.cc:237] T b284ed2cb0264ced99a4ffd90983d902 P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { replace: true } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false } }
I20240501 07:59:11.979599 6577 raft_consensus.cc:1261] T b284ed2cb0264ced99a4ffd90983d902 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Refusing update from remote peer 1f9cb72be98941b5a8e504cbe34132b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:59:11.979683 6311 raft_consensus.cc:1261] T b284ed2cb0264ced99a4ffd90983d902 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Refusing update from remote peer 1f9cb72be98941b5a8e504cbe34132b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:59:11.979707 8924 raft_consensus.cc:1261] T b284ed2cb0264ced99a4ffd90983d902 P 13d05ebdca8c4f489c597ce550dd773d [term 1 LEARNER]: Refusing update from remote peer 1f9cb72be98941b5a8e504cbe34132b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:59:11.979771 10753 ts_tablet_manager.cc:1888] T e92d6aeeb3df4ea3852c6ebbed0e07c5 P 3b164cbbc00f4e8eb3254ddd349a3ff9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 07:59:11.979838 10753 log.cc:1196] T e92d6aeeb3df4ea3852c6ebbed0e07c5 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/e92d6aeeb3df4ea3852c6ebbed0e07c5
I20240501 07:59:11.980018 10324 consensus_queue.cc:1035] T b284ed2cb0264ced99a4ffd90983d902 P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:59:11.980175 9618 consensus_queue.cc:1035] T b284ed2cb0264ced99a4ffd90983d902 P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:59:11.980428 6160 catalog_manager.cc:4830] TS 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131:34471): tablet e92d6aeeb3df4ea3852c6ebbed0e07c5 (table rebalance_test_table_13 [id=886e8147658c4cffb480ca3fb821c563]) successfully deleted
I20240501 07:59:11.980713 9990 consensus_queue.cc:1035] T b284ed2cb0264ced99a4ffd90983d902 P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:59:11.982666 9990 raft_consensus.cc:2932] T b284ed2cb0264ced99a4ffd90983d902 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { replace: true } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false } } }
I20240501 07:59:11.983909 6313 raft_consensus.cc:2932] T b284ed2cb0264ced99a4ffd90983d902 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { replace: true } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false } } }
I20240501 07:59:11.984843 8924 raft_consensus.cc:2932] T b284ed2cb0264ced99a4ffd90983d902 P 13d05ebdca8c4f489c597ce550dd773d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { replace: true } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false } } }
I20240501 07:59:11.987376 6171 catalog_manager.cc:5484] T b284ed2cb0264ced99a4ffd90983d902 P 1f9cb72be98941b5a8e504cbe34132b9 reported cstate change: config changed from index 4 to 5, 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1f9cb72be98941b5a8e504cbe34132b9" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 07:59:11.988642 6577 raft_consensus.cc:2932] T b284ed2cb0264ced99a4ffd90983d902 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { replace: true } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false } } }
I20240501 07:59:11.990697 6707 consensus_queue.cc:237] T b284ed2cb0264ced99a4ffd90983d902 P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false } }
I20240501 07:59:11.992601 8924 raft_consensus.cc:1261] T b284ed2cb0264ced99a4ffd90983d902 P 13d05ebdca8c4f489c597ce550dd773d [term 1 FOLLOWER]: Refusing update from remote peer 1f9cb72be98941b5a8e504cbe34132b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 07:59:11.992705 6577 raft_consensus.cc:1261] T b284ed2cb0264ced99a4ffd90983d902 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Refusing update from remote peer 1f9cb72be98941b5a8e504cbe34132b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 07:59:11.993129 9618 consensus_queue.cc:1035] T b284ed2cb0264ced99a4ffd90983d902 P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 07:59:11.993497 9618 consensus_queue.cc:1035] T b284ed2cb0264ced99a4ffd90983d902 P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 07:59:11.995074 10324 raft_consensus.cc:2932] T b284ed2cb0264ced99a4ffd90983d902 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false } } }
I20240501 07:59:11.995482 8924 raft_consensus.cc:2932] T b284ed2cb0264ced99a4ffd90983d902 P 13d05ebdca8c4f489c597ce550dd773d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false } } }
I20240501 07:59:11.995481 6577 raft_consensus.cc:2932] T b284ed2cb0264ced99a4ffd90983d902 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false } } }
I20240501 07:59:11.996778 6159 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b284ed2cb0264ced99a4ffd90983d902 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 07:59:11.998255 6172 catalog_manager.cc:5484] T b284ed2cb0264ced99a4ffd90983d902 P 3b164cbbc00f4e8eb3254ddd349a3ff9 reported cstate change: config changed from index 5 to 6, VOTER 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "1f9cb72be98941b5a8e504cbe34132b9" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false } } }
I20240501 07:59:12.002384 6294 tablet_service.cc:1514] Processing DeleteTablet for tablet b284ed2cb0264ced99a4ffd90983d902 with delete_type TABLET_DATA_TOMBSTONED (TS 8b8b3ad3aa8f4053b2167dd550f2086f not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:45480
I20240501 07:59:12.002828 10309 tablet_replica.cc:330] T b284ed2cb0264ced99a4ffd90983d902 P 8b8b3ad3aa8f4053b2167dd550f2086f: stopping tablet replica
I20240501 07:59:12.002971 10309 raft_consensus.cc:2229] T b284ed2cb0264ced99a4ffd90983d902 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:12.003157 10309 raft_consensus.cc:2258] T b284ed2cb0264ced99a4ffd90983d902 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:12.003895 10309 ts_tablet_manager.cc:1875] T b284ed2cb0264ced99a4ffd90983d902 P 8b8b3ad3aa8f4053b2167dd550f2086f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:59:12.006001 10309 ts_tablet_manager.cc:1888] T b284ed2cb0264ced99a4ffd90983d902 P 8b8b3ad3aa8f4053b2167dd550f2086f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 07:59:12.006155 10309 log.cc:1196] T b284ed2cb0264ced99a4ffd90983d902 P 8b8b3ad3aa8f4053b2167dd550f2086f: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/b284ed2cb0264ced99a4ffd90983d902
I20240501 07:59:12.006785 6157 catalog_manager.cc:4830] TS 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129:46875): tablet b284ed2cb0264ced99a4ffd90983d902 (table rebalance_test_table_18 [id=25ecaaa6eafe48ecb3f3ff353a6d1170]) successfully deleted
I20240501 07:59:12.049111 6164 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55950:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 07:59:12.054812 11146 tablet.cc:1722] T 1eae52e7f97f45b48697c624192aa14b P 83f20e84af4c40d09a4804b6b26e697f: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 07:59:12.055166 11146 tablet.cc:1620] T 1eae52e7f97f45b48697c624192aa14b P 83f20e84af4c40d09a4804b6b26e697f: MemRowSet was empty: no flush needed.
I20240501 07:59:12.055116 11148 tablet.cc:1722] T 1eae52e7f97f45b48697c624192aa14b P 1f9cb72be98941b5a8e504cbe34132b9: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 07:59:12.055420 11148 tablet.cc:1620] T 1eae52e7f97f45b48697c624192aa14b P 1f9cb72be98941b5a8e504cbe34132b9: MemRowSet was empty: no flush needed.
I20240501 07:59:12.057052 11147 tablet.cc:1722] T 1eae52e7f97f45b48697c624192aa14b P 74077b61d6f54342bf9736b4a9f2eade: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 07:59:12.057368 11147 tablet.cc:1620] T 1eae52e7f97f45b48697c624192aa14b P 74077b61d6f54342bf9736b4a9f2eade: MemRowSet was empty: no flush needed.
I20240501 07:59:12.061255 9400 raft_consensus.cc:1050] T fb6b46072a8d46e18b53b9f8de702859 P 13d05ebdca8c4f489c597ce550dd773d: attempting to promote NON_VOTER 8b8b3ad3aa8f4053b2167dd550f2086f to VOTER
I20240501 07:59:12.061797 9400 consensus_queue.cc:237] T fb6b46072a8d46e18b53b9f8de702859 P 13d05ebdca8c4f489c597ce550dd773d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { replace: true } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { promote: false } }
I20240501 07:59:12.064157 8794 raft_consensus.cc:1261] T fb6b46072a8d46e18b53b9f8de702859 P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Refusing update from remote peer 13d05ebdca8c4f489c597ce550dd773d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 07:59:12.064239 6311 raft_consensus.cc:1261] T fb6b46072a8d46e18b53b9f8de702859 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 LEARNER]: Refusing update from remote peer 13d05ebdca8c4f489c597ce550dd773d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 07:59:12.065327 11215 consensus_queue.cc:1035] T fb6b46072a8d46e18b53b9f8de702859 P 13d05ebdca8c4f489c597ce550dd773d [LEADER]: Connected to new peer: Peer: permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 07:59:12.065495 9189 raft_consensus.cc:1261] T fb6b46072a8d46e18b53b9f8de702859 P 83f20e84af4c40d09a4804b6b26e697f [term 1 FOLLOWER]: Refusing update from remote peer 13d05ebdca8c4f489c597ce550dd773d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 07:59:12.066102 10478 consensus_queue.cc:1035] T fb6b46072a8d46e18b53b9f8de702859 P 13d05ebdca8c4f489c597ce550dd773d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 07:59:12.066217 6168 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 0bc706e85ce74cef97a4827ea4d4aa65: rebalancer_dynamic_table [id=a57717a5df61497abe35bc282984a9ab] alter complete (version 2)
I20240501 07:59:12.066429 10478 consensus_queue.cc:1035] T fb6b46072a8d46e18b53b9f8de702859 P 13d05ebdca8c4f489c597ce550dd773d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 07:59:12.072465 9400 raft_consensus.cc:2932] T fb6b46072a8d46e18b53b9f8de702859 P 13d05ebdca8c4f489c597ce550dd773d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { replace: true } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { promote: false } } }
I20240501 07:59:12.072876 6311 raft_consensus.cc:2932] T fb6b46072a8d46e18b53b9f8de702859 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { replace: true } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { promote: false } } }
I20240501 07:59:12.075440 8794 raft_consensus.cc:2932] T fb6b46072a8d46e18b53b9f8de702859 P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { replace: true } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { promote: false } } }
I20240501 07:59:12.075735 6168 catalog_manager.cc:5484] T fb6b46072a8d46e18b53b9f8de702859 P 8b8b3ad3aa8f4053b2167dd550f2086f reported cstate change: config changed from index 2 to 3, 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "13d05ebdca8c4f489c597ce550dd773d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { replace: true } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { promote: false } } }
I20240501 07:59:12.078755 9189 raft_consensus.cc:2932] T fb6b46072a8d46e18b53b9f8de702859 P 83f20e84af4c40d09a4804b6b26e697f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { replace: true } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { promote: false } } }
I20240501 07:59:12.083230 8924 consensus_queue.cc:237] T fb6b46072a8d46e18b53b9f8de702859 P 13d05ebdca8c4f489c597ce550dd773d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { promote: false } }
I20240501 07:59:12.084414 9189 raft_consensus.cc:1261] T fb6b46072a8d46e18b53b9f8de702859 P 83f20e84af4c40d09a4804b6b26e697f [term 1 FOLLOWER]: Refusing update from remote peer 13d05ebdca8c4f489c597ce550dd773d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:59:12.084414 6311 raft_consensus.cc:1261] T fb6b46072a8d46e18b53b9f8de702859 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Refusing update from remote peer 13d05ebdca8c4f489c597ce550dd773d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:59:12.084767 10235 consensus_queue.cc:1035] T fb6b46072a8d46e18b53b9f8de702859 P 13d05ebdca8c4f489c597ce550dd773d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:12.084985 11215 consensus_queue.cc:1035] T fb6b46072a8d46e18b53b9f8de702859 P 13d05ebdca8c4f489c597ce550dd773d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:12.086617 10235 raft_consensus.cc:2932] T fb6b46072a8d46e18b53b9f8de702859 P 13d05ebdca8c4f489c597ce550dd773d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { promote: false } } }
I20240501 07:59:12.087160 6311 raft_consensus.cc:2932] T fb6b46072a8d46e18b53b9f8de702859 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { promote: false } } }
I20240501 07:59:12.087268 9189 raft_consensus.cc:2932] T fb6b46072a8d46e18b53b9f8de702859 P 83f20e84af4c40d09a4804b6b26e697f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { promote: false } } }
I20240501 07:59:12.088940 6159 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet fb6b46072a8d46e18b53b9f8de702859 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 07:59:12.089893 6172 catalog_manager.cc:5484] T fb6b46072a8d46e18b53b9f8de702859 P 13d05ebdca8c4f489c597ce550dd773d reported cstate change: config changed from index 3 to 4, VOTER 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133) evicted. New cstate: current_term: 1 leader_uuid: "13d05ebdca8c4f489c597ce550dd773d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 07:59:12.092447 11085 raft_consensus.cc:1050] T f68dbc1f65eb4f1b91a62e2de82a7676 P 8b8b3ad3aa8f4053b2167dd550f2086f: attempting to promote NON_VOTER 83f20e84af4c40d09a4804b6b26e697f to VOTER
I20240501 07:59:12.092869 11085 consensus_queue.cc:237] T f68dbc1f65eb4f1b91a62e2de82a7676 P 8b8b3ad3aa8f4053b2167dd550f2086f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false } }
I20240501 07:59:12.094158 8774 tablet_service.cc:1514] Processing DeleteTablet for tablet fb6b46072a8d46e18b53b9f8de702859 with delete_type TABLET_DATA_TOMBSTONED (TS 74077b61d6f54342bf9736b4a9f2eade not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:43136
I20240501 07:59:12.094300 9189 raft_consensus.cc:1261] T f68dbc1f65eb4f1b91a62e2de82a7676 P 83f20e84af4c40d09a4804b6b26e697f [term 1 LEARNER]: Refusing update from remote peer 8b8b3ad3aa8f4053b2167dd550f2086f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:59:12.094527 6444 raft_consensus.cc:1261] T f68dbc1f65eb4f1b91a62e2de82a7676 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Refusing update from remote peer 8b8b3ad3aa8f4053b2167dd550f2086f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:59:12.094873 9373 consensus_queue.cc:1035] T f68dbc1f65eb4f1b91a62e2de82a7676 P 8b8b3ad3aa8f4053b2167dd550f2086f [LEADER]: Connected to new peer: Peer: permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:59:12.094992 6577 raft_consensus.cc:1261] T f68dbc1f65eb4f1b91a62e2de82a7676 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Refusing update from remote peer 8b8b3ad3aa8f4053b2167dd550f2086f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:59:12.095221 9373 consensus_queue.cc:1035] T f68dbc1f65eb4f1b91a62e2de82a7676 P 8b8b3ad3aa8f4053b2167dd550f2086f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:59:12.095417 9964 consensus_queue.cc:1035] T f68dbc1f65eb4f1b91a62e2de82a7676 P 8b8b3ad3aa8f4053b2167dd550f2086f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:59:12.095863 11226 tablet_replica.cc:330] T fb6b46072a8d46e18b53b9f8de702859 P 74077b61d6f54342bf9736b4a9f2eade: stopping tablet replica
I20240501 07:59:12.096028 11226 raft_consensus.cc:2229] T fb6b46072a8d46e18b53b9f8de702859 P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:12.096197 11226 raft_consensus.cc:2258] T fb6b46072a8d46e18b53b9f8de702859 P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:12.096899 11226 ts_tablet_manager.cc:1875] T fb6b46072a8d46e18b53b9f8de702859 P 74077b61d6f54342bf9736b4a9f2eade: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:59:12.097608 9964 raft_consensus.cc:2932] T f68dbc1f65eb4f1b91a62e2de82a7676 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false } } }
I20240501 07:59:12.098867 6577 raft_consensus.cc:2932] T f68dbc1f65eb4f1b91a62e2de82a7676 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false } } }
I20240501 07:59:12.099898 11226 ts_tablet_manager.cc:1888] T fb6b46072a8d46e18b53b9f8de702859 P 74077b61d6f54342bf9736b4a9f2eade: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 07:59:12.099975 11226 log.cc:1196] T fb6b46072a8d46e18b53b9f8de702859 P 74077b61d6f54342bf9736b4a9f2eade: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/fb6b46072a8d46e18b53b9f8de702859
I20240501 07:59:12.099882 6444 raft_consensus.cc:2932] T f68dbc1f65eb4f1b91a62e2de82a7676 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false } } }
I20240501 07:59:12.101279 6157 catalog_manager.cc:4830] TS 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133:39859): tablet fb6b46072a8d46e18b53b9f8de702859 (table rebalancer_extra_table_0 [id=9ec2540a0f4b49ada014e0c68ec3b2a0]) successfully deleted
I20240501 07:59:12.102015 6164 catalog_manager.cc:5484] T f68dbc1f65eb4f1b91a62e2de82a7676 P 3b164cbbc00f4e8eb3254ddd349a3ff9 reported cstate change: config changed from index 4 to 5, 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false } } }
I20240501 07:59:12.103271 9188 raft_consensus.cc:1203] T aa1b8716228b4fd9af95870b7a613fb2 P 83f20e84af4c40d09a4804b6b26e697f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 07:59:12.104313 9189 raft_consensus.cc:2932] T f68dbc1f65eb4f1b91a62e2de82a7676 P 83f20e84af4c40d09a4804b6b26e697f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false } } }
I20240501 07:59:12.107517 6311 consensus_queue.cc:237] T f68dbc1f65eb4f1b91a62e2de82a7676 P 8b8b3ad3aa8f4053b2167dd550f2086f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false } }
I20240501 07:59:12.109030 6577 raft_consensus.cc:1261] T f68dbc1f65eb4f1b91a62e2de82a7676 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Refusing update from remote peer 8b8b3ad3aa8f4053b2167dd550f2086f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 07:59:12.109510 10867 consensus_queue.cc:1035] T f68dbc1f65eb4f1b91a62e2de82a7676 P 8b8b3ad3aa8f4053b2167dd550f2086f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 07:59:12.110409 9189 raft_consensus.cc:1261] T f68dbc1f65eb4f1b91a62e2de82a7676 P 83f20e84af4c40d09a4804b6b26e697f [term 1 FOLLOWER]: Refusing update from remote peer 8b8b3ad3aa8f4053b2167dd550f2086f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 07:59:12.111152 11085 raft_consensus.cc:2932] T f68dbc1f65eb4f1b91a62e2de82a7676 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false } } }
I20240501 07:59:12.111531 9373 consensus_queue.cc:1035] T f68dbc1f65eb4f1b91a62e2de82a7676 P 8b8b3ad3aa8f4053b2167dd550f2086f [LEADER]: Connected to new peer: Peer: permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 07:59:12.111671 6577 raft_consensus.cc:2932] T f68dbc1f65eb4f1b91a62e2de82a7676 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false } } }
I20240501 07:59:12.113080 6157 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f68dbc1f65eb4f1b91a62e2de82a7676 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 07:59:12.113893 9189 raft_consensus.cc:2932] T f68dbc1f65eb4f1b91a62e2de82a7676 P 83f20e84af4c40d09a4804b6b26e697f [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false } } }
I20240501 07:59:12.113976 6164 catalog_manager.cc:5484] T f68dbc1f65eb4f1b91a62e2de82a7676 P 8b8b3ad3aa8f4053b2167dd550f2086f reported cstate change: config changed from index 5 to 6, VOTER 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 07:59:12.117064 6424 tablet_service.cc:1514] Processing DeleteTablet for tablet f68dbc1f65eb4f1b91a62e2de82a7676 with delete_type TABLET_DATA_TOMBSTONED (TS 8315d22c56f7439a9e1fc82f153f018c not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:45034
I20240501 07:59:12.117384 10751 tablet_replica.cc:330] T f68dbc1f65eb4f1b91a62e2de82a7676 P 8315d22c56f7439a9e1fc82f153f018c: stopping tablet replica
I20240501 07:59:12.117556 10751 raft_consensus.cc:2229] T f68dbc1f65eb4f1b91a62e2de82a7676 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:12.117736 10751 raft_consensus.cc:2258] T f68dbc1f65eb4f1b91a62e2de82a7676 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:12.118412 10751 ts_tablet_manager.cc:1875] T f68dbc1f65eb4f1b91a62e2de82a7676 P 8315d22c56f7439a9e1fc82f153f018c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:59:12.120697 10751 ts_tablet_manager.cc:1888] T f68dbc1f65eb4f1b91a62e2de82a7676 P 8315d22c56f7439a9e1fc82f153f018c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 07:59:12.120776 10751 log.cc:1196] T f68dbc1f65eb4f1b91a62e2de82a7676 P 8315d22c56f7439a9e1fc82f153f018c: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/f68dbc1f65eb4f1b91a62e2de82a7676
I20240501 07:59:12.121814 6159 catalog_manager.cc:4830] TS 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130:32941): tablet f68dbc1f65eb4f1b91a62e2de82a7676 (table rebalance_test_table_5 [id=336b9f044a794b6a8a51d76e720b0ecd]) successfully deleted
I20240501 07:59:12.134681 6444 consensus_queue.cc:237] T df00e37199e944589a0a5f3feeee308a P 8315d22c56f7439a9e1fc82f153f018c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: true } }
I20240501 07:59:12.136049 6577 raft_consensus.cc:1261] T df00e37199e944589a0a5f3feeee308a P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Refusing update from remote peer 8315d22c56f7439a9e1fc82f153f018c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 07:59:12.136274 6381 consensus_peers.cc:485] T df00e37199e944589a0a5f3feeee308a P 8315d22c56f7439a9e1fc82f153f018c -> Peer 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134:40141): Couldn't send request to peer 13d05ebdca8c4f489c597ce550dd773d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: df00e37199e944589a0a5f3feeee308a. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:12.136296 6707 raft_consensus.cc:1261] T df00e37199e944589a0a5f3feeee308a P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Refusing update from remote peer 8315d22c56f7439a9e1fc82f153f018c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:59:12.136646 9116 consensus_queue.cc:1035] T df00e37199e944589a0a5f3feeee308a P 8315d22c56f7439a9e1fc82f153f018c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:12.136838 10111 consensus_queue.cc:1035] T df00e37199e944589a0a5f3feeee308a P 8315d22c56f7439a9e1fc82f153f018c [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:12.138660 9489 raft_consensus.cc:2932] T df00e37199e944589a0a5f3feeee308a P 8315d22c56f7439a9e1fc82f153f018c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: true } } }
I20240501 07:59:12.139042 6577 raft_consensus.cc:2932] T df00e37199e944589a0a5f3feeee308a P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: true } } }
I20240501 07:59:12.141822 6707 raft_consensus.cc:2932] T df00e37199e944589a0a5f3feeee308a P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: true } } }
I20240501 07:59:12.142138 6168 catalog_manager.cc:5484] T df00e37199e944589a0a5f3feeee308a P 8315d22c56f7439a9e1fc82f153f018c reported cstate change: config changed from index -1 to 4, NON_VOTER 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "8315d22c56f7439a9e1fc82f153f018c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 07:59:12.149839 6577 consensus_queue.cc:237] T d3f2250a62c6429e886753aaffcbe883 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { replace: true } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: true } }
I20240501 07:59:12.151587 9189 raft_consensus.cc:1261] T d3f2250a62c6429e886753aaffcbe883 P 83f20e84af4c40d09a4804b6b26e697f [term 1 FOLLOWER]: Refusing update from remote peer 3b164cbbc00f4e8eb3254ddd349a3ff9: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 07:59:12.151752 9940 consensus_queue.cc:1035] T 1de93edea7714491ae339db8cdfec090 P a505e7cb133c4392bfda8a4a20874ae9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 07:59:12.152334 6511 consensus_peers.cc:485] T d3f2250a62c6429e886753aaffcbe883 P 3b164cbbc00f4e8eb3254ddd349a3ff9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d3f2250a62c6429e886753aaffcbe883. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:12.152518 9507 consensus_queue.cc:1035] T d3f2250a62c6429e886753aaffcbe883 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 07:59:12.152701 6707 raft_consensus.cc:1261] T d3f2250a62c6429e886753aaffcbe883 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Refusing update from remote peer 3b164cbbc00f4e8eb3254ddd349a3ff9: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 07:59:12.153082 9507 consensus_queue.cc:1035] T d3f2250a62c6429e886753aaffcbe883 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 07:59:12.155267 10945 consensus_queue.cc:1035] T 1de93edea7714491ae339db8cdfec090 P a505e7cb133c4392bfda8a4a20874ae9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:12.155822 9507 raft_consensus.cc:2932] T d3f2250a62c6429e886753aaffcbe883 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { replace: true } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: true } } }
I20240501 07:59:12.158514 6164 catalog_manager.cc:5484] T d3f2250a62c6429e886753aaffcbe883 P 3b164cbbc00f4e8eb3254ddd349a3ff9 reported cstate change: config changed from index 6 to 7, NON_VOTER a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 07:59:12.159402 6707 raft_consensus.cc:2932] T d3f2250a62c6429e886753aaffcbe883 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { replace: true } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: true } } }
I20240501 07:59:12.162307 9189 raft_consensus.cc:2932] T d3f2250a62c6429e886753aaffcbe883 P 83f20e84af4c40d09a4804b6b26e697f [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { replace: true } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: true } } }
I20240501 07:59:12.173122 6168 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55950:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 07:59:12.176375 11272 ts_tablet_manager.cc:902] T 72aa4dc7664b48b7a73d34feba7d44d3 P 8315d22c56f7439a9e1fc82f153f018c: Initiating tablet copy from peer 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133:39859)
I20240501 07:59:12.177428 11272 tablet_copy_client.cc:308] T 72aa4dc7664b48b7a73d34feba7d44d3 P 8315d22c56f7439a9e1fc82f153f018c: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.133:39859
I20240501 07:59:12.178026 8814 tablet_copy_service.cc:140] P 74077b61d6f54342bf9736b4a9f2eade: Received BeginTabletCopySession request for tablet 72aa4dc7664b48b7a73d34feba7d44d3 from peer 8315d22c56f7439a9e1fc82f153f018c ({username='slave'} at 127.0.99.130:51123)
I20240501 07:59:12.178141 8814 tablet_copy_service.cc:161] P 74077b61d6f54342bf9736b4a9f2eade: Beginning new tablet copy session on tablet 72aa4dc7664b48b7a73d34feba7d44d3 from peer 8315d22c56f7439a9e1fc82f153f018c at {username='slave'} at 127.0.99.130:51123: session id = 8315d22c56f7439a9e1fc82f153f018c-72aa4dc7664b48b7a73d34feba7d44d3
I20240501 07:59:12.179198 8814 tablet_copy_source_session.cc:205] T 72aa4dc7664b48b7a73d34feba7d44d3 P 74077b61d6f54342bf9736b4a9f2eade: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:12.179404 11146 tablet.cc:1722] T 1eae52e7f97f45b48697c624192aa14b P 83f20e84af4c40d09a4804b6b26e697f: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 07:59:12.180493 11272 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72aa4dc7664b48b7a73d34feba7d44d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:12.180945 11147 tablet.cc:1722] T 1eae52e7f97f45b48697c624192aa14b P 74077b61d6f54342bf9736b4a9f2eade: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 07:59:12.185351 11148 tablet.cc:1722] T 1eae52e7f97f45b48697c624192aa14b P 1f9cb72be98941b5a8e504cbe34132b9: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 07:59:12.186265 6168 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 0bc706e85ce74cef97a4827ea4d4aa65: rebalancer_dynamic_table_new_name [id=a57717a5df61497abe35bc282984a9ab] alter complete (version 3)
I20240501 07:59:12.188161 11272 tablet_copy_client.cc:783] T 72aa4dc7664b48b7a73d34feba7d44d3 P 8315d22c56f7439a9e1fc82f153f018c: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:12.188364 11272 tablet_copy_client.cc:647] T 72aa4dc7664b48b7a73d34feba7d44d3 P 8315d22c56f7439a9e1fc82f153f018c: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:12.189739 11272 tablet_copy_client.cc:523] T 72aa4dc7664b48b7a73d34feba7d44d3 P 8315d22c56f7439a9e1fc82f153f018c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:12.191325 11272 tablet_bootstrap.cc:492] T 72aa4dc7664b48b7a73d34feba7d44d3 P 8315d22c56f7439a9e1fc82f153f018c: Bootstrap starting.
I20240501 07:59:12.196273 11272 tablet_bootstrap.cc:492] T 72aa4dc7664b48b7a73d34feba7d44d3 P 8315d22c56f7439a9e1fc82f153f018c: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:12.196568 11272 tablet_bootstrap.cc:492] T 72aa4dc7664b48b7a73d34feba7d44d3 P 8315d22c56f7439a9e1fc82f153f018c: Bootstrap complete.
I20240501 07:59:12.196728 11272 ts_tablet_manager.cc:1367] T 72aa4dc7664b48b7a73d34feba7d44d3 P 8315d22c56f7439a9e1fc82f153f018c: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20240501 07:59:12.197494 11272 raft_consensus.cc:348] T 72aa4dc7664b48b7a73d34feba7d44d3 P 8315d22c56f7439a9e1fc82f153f018c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { replace: true } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { promote: true } }
I20240501 07:59:12.197628 11272 raft_consensus.cc:729] T 72aa4dc7664b48b7a73d34feba7d44d3 P 8315d22c56f7439a9e1fc82f153f018c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8315d22c56f7439a9e1fc82f153f018c, State: Initialized, Role: LEARNER
I20240501 07:59:12.197733 11272 consensus_queue.cc:260] T 72aa4dc7664b48b7a73d34feba7d44d3 P 8315d22c56f7439a9e1fc82f153f018c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { replace: true } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { promote: true } }
I20240501 07:59:12.198882 11272 ts_tablet_manager.cc:1398] T 72aa4dc7664b48b7a73d34feba7d44d3 P 8315d22c56f7439a9e1fc82f153f018c: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:59:12.199064 11283 ts_tablet_manager.cc:902] T 2976d6353e8947b7b6c37f72efbeeac4 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Initiating tablet copy from peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149)
I20240501 07:59:12.199863 11283 tablet_copy_client.cc:308] T 2976d6353e8947b7b6c37f72efbeeac4 P 3b164cbbc00f4e8eb3254ddd349a3ff9: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.136:46149
I20240501 07:59:12.199854 8814 tablet_copy_service.cc:342] P 74077b61d6f54342bf9736b4a9f2eade: Request end of tablet copy session 8315d22c56f7439a9e1fc82f153f018c-72aa4dc7664b48b7a73d34feba7d44d3 received from {username='slave'} at 127.0.99.130:51123
I20240501 07:59:12.199934 8814 tablet_copy_service.cc:433] P 74077b61d6f54342bf9736b4a9f2eade: ending tablet copy session 8315d22c56f7439a9e1fc82f153f018c-72aa4dc7664b48b7a73d34feba7d44d3 on tablet 72aa4dc7664b48b7a73d34feba7d44d3 with peer 8315d22c56f7439a9e1fc82f153f018c
I20240501 07:59:12.200531 9207 tablet_copy_service.cc:140] P 83f20e84af4c40d09a4804b6b26e697f: Received BeginTabletCopySession request for tablet 2976d6353e8947b7b6c37f72efbeeac4 from peer 3b164cbbc00f4e8eb3254ddd349a3ff9 ({username='slave'} at 127.0.99.131:33477)
I20240501 07:59:12.200664 9207 tablet_copy_service.cc:161] P 83f20e84af4c40d09a4804b6b26e697f: Beginning new tablet copy session on tablet 2976d6353e8947b7b6c37f72efbeeac4 from peer 3b164cbbc00f4e8eb3254ddd349a3ff9 at {username='slave'} at 127.0.99.131:33477: session id = 3b164cbbc00f4e8eb3254ddd349a3ff9-2976d6353e8947b7b6c37f72efbeeac4
I20240501 07:59:12.201982 9207 tablet_copy_source_session.cc:205] T 2976d6353e8947b7b6c37f72efbeeac4 P 83f20e84af4c40d09a4804b6b26e697f: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:12.203294 11283 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2976d6353e8947b7b6c37f72efbeeac4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:12.205588 11283 tablet_copy_client.cc:783] T 2976d6353e8947b7b6c37f72efbeeac4 P 3b164cbbc00f4e8eb3254ddd349a3ff9: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:12.205783 11283 tablet_copy_client.cc:647] T 2976d6353e8947b7b6c37f72efbeeac4 P 3b164cbbc00f4e8eb3254ddd349a3ff9: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:12.208182 11283 tablet_copy_client.cc:523] T 2976d6353e8947b7b6c37f72efbeeac4 P 3b164cbbc00f4e8eb3254ddd349a3ff9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:12.210132 11283 tablet_bootstrap.cc:492] T 2976d6353e8947b7b6c37f72efbeeac4 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Bootstrap starting.
I20240501 07:59:12.211402 11085 raft_consensus.cc:1050] T 2e540f389b2a4aef93a020f549d11b67 P 8b8b3ad3aa8f4053b2167dd550f2086f: attempting to promote NON_VOTER 83f20e84af4c40d09a4804b6b26e697f to VOTER
I20240501 07:59:12.211915 11085 consensus_queue.cc:237] T 2e540f389b2a4aef93a020f549d11b67 P 8b8b3ad3aa8f4053b2167dd550f2086f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false } }
I20240501 07:59:12.213591 6444 raft_consensus.cc:1261] T 2e540f389b2a4aef93a020f549d11b67 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Refusing update from remote peer 8b8b3ad3aa8f4053b2167dd550f2086f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:59:12.214288 6577 raft_consensus.cc:1261] T 2e540f389b2a4aef93a020f549d11b67 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Refusing update from remote peer 8b8b3ad3aa8f4053b2167dd550f2086f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:59:12.214862 9964 consensus_queue.cc:1035] T 2e540f389b2a4aef93a020f549d11b67 P 8b8b3ad3aa8f4053b2167dd550f2086f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:59:12.215909 9189 raft_consensus.cc:1261] T 2e540f389b2a4aef93a020f549d11b67 P 83f20e84af4c40d09a4804b6b26e697f [term 1 LEARNER]: Refusing update from remote peer 8b8b3ad3aa8f4053b2167dd550f2086f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:59:12.215938 11283 tablet_bootstrap.cc:492] T 2976d6353e8947b7b6c37f72efbeeac4 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:12.216241 11283 tablet_bootstrap.cc:492] T 2976d6353e8947b7b6c37f72efbeeac4 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Bootstrap complete.
I20240501 07:59:12.216413 11283 ts_tablet_manager.cc:1367] T 2976d6353e8947b7b6c37f72efbeeac4 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.003s
I20240501 07:59:12.216562 11094 consensus_queue.cc:1035] T 2e540f389b2a4aef93a020f549d11b67 P 8b8b3ad3aa8f4053b2167dd550f2086f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:59:12.216877 11283 raft_consensus.cc:348] T 2976d6353e8947b7b6c37f72efbeeac4 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: true } }
I20240501 07:59:12.217044 11283 raft_consensus.cc:729] T 2976d6353e8947b7b6c37f72efbeeac4 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3b164cbbc00f4e8eb3254ddd349a3ff9, State: Initialized, Role: LEARNER
I20240501 07:59:12.217211 11283 consensus_queue.cc:260] T 2976d6353e8947b7b6c37f72efbeeac4 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: true } }
I20240501 07:59:12.217811 9964 consensus_queue.cc:1035] T 2e540f389b2a4aef93a020f549d11b67 P 8b8b3ad3aa8f4053b2167dd550f2086f [LEADER]: Connected to new peer: Peer: permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:59:12.218178 11283 ts_tablet_manager.cc:1398] T 2976d6353e8947b7b6c37f72efbeeac4 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:59:12.218698 9207 tablet_copy_service.cc:342] P 83f20e84af4c40d09a4804b6b26e697f: Request end of tablet copy session 3b164cbbc00f4e8eb3254ddd349a3ff9-2976d6353e8947b7b6c37f72efbeeac4 received from {username='slave'} at 127.0.99.131:33477
I20240501 07:59:12.218789 9207 tablet_copy_service.cc:433] P 83f20e84af4c40d09a4804b6b26e697f: ending tablet copy session 3b164cbbc00f4e8eb3254ddd349a3ff9-2976d6353e8947b7b6c37f72efbeeac4 on tablet 2976d6353e8947b7b6c37f72efbeeac4 with peer 3b164cbbc00f4e8eb3254ddd349a3ff9
I20240501 07:59:12.222911 11094 raft_consensus.cc:2932] T 2e540f389b2a4aef93a020f549d11b67 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false } } }
I20240501 07:59:12.223932 6577 raft_consensus.cc:2932] T 2e540f389b2a4aef93a020f549d11b67 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false } } }
I20240501 07:59:12.225658 9189 raft_consensus.cc:2932] T 2e540f389b2a4aef93a020f549d11b67 P 83f20e84af4c40d09a4804b6b26e697f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false } } }
I20240501 07:59:12.225659 6444 raft_consensus.cc:2932] T 2e540f389b2a4aef93a020f549d11b67 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false } } }
I20240501 07:59:12.228823 6172 catalog_manager.cc:5484] T 2e540f389b2a4aef93a020f549d11b67 P 83f20e84af4c40d09a4804b6b26e697f reported cstate change: config changed from index 4 to 5, 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false } } }
I20240501 07:59:12.233079 6311 consensus_queue.cc:237] T 2e540f389b2a4aef93a020f549d11b67 P 8b8b3ad3aa8f4053b2167dd550f2086f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false } }
I20240501 07:59:12.234403 6577 raft_consensus.cc:1261] T 2e540f389b2a4aef93a020f549d11b67 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Refusing update from remote peer 8b8b3ad3aa8f4053b2167dd550f2086f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 07:59:12.234493 9189 raft_consensus.cc:1261] T 2e540f389b2a4aef93a020f549d11b67 P 83f20e84af4c40d09a4804b6b26e697f [term 1 FOLLOWER]: Refusing update from remote peer 8b8b3ad3aa8f4053b2167dd550f2086f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 07:59:12.234824 9964 consensus_queue.cc:1035] T 2e540f389b2a4aef93a020f549d11b67 P 8b8b3ad3aa8f4053b2167dd550f2086f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 07:59:12.235180 11085 consensus_queue.cc:1035] T 2e540f389b2a4aef93a020f549d11b67 P 8b8b3ad3aa8f4053b2167dd550f2086f [LEADER]: Connected to new peer: Peer: permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 07:59:12.236851 11085 raft_consensus.cc:2932] T 2e540f389b2a4aef93a020f549d11b67 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false } } }
I20240501 07:59:12.238531 6157 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 2e540f389b2a4aef93a020f549d11b67 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 07:59:12.238955 9189 raft_consensus.cc:2932] T 2e540f389b2a4aef93a020f549d11b67 P 83f20e84af4c40d09a4804b6b26e697f [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false } } }
I20240501 07:59:12.239887 6577 raft_consensus.cc:2932] T 2e540f389b2a4aef93a020f549d11b67 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false } } }
I20240501 07:59:12.241899 6170 catalog_manager.cc:5484] T 2e540f389b2a4aef93a020f549d11b67 P 8b8b3ad3aa8f4053b2167dd550f2086f reported cstate change: config changed from index 5 to 6, VOTER 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 07:59:12.244752 6424 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e540f389b2a4aef93a020f549d11b67 with delete_type TABLET_DATA_TOMBSTONED (TS 8315d22c56f7439a9e1fc82f153f018c not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:45034
I20240501 07:59:12.245055 10751 tablet_replica.cc:330] T 2e540f389b2a4aef93a020f549d11b67 P 8315d22c56f7439a9e1fc82f153f018c: stopping tablet replica
I20240501 07:59:12.245198 10751 raft_consensus.cc:2229] T 2e540f389b2a4aef93a020f549d11b67 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:12.245364 10751 raft_consensus.cc:2258] T 2e540f389b2a4aef93a020f549d11b67 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:12.246182 10751 ts_tablet_manager.cc:1875] T 2e540f389b2a4aef93a020f549d11b67 P 8315d22c56f7439a9e1fc82f153f018c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:59:12.248551 10751 ts_tablet_manager.cc:1888] T 2e540f389b2a4aef93a020f549d11b67 P 8315d22c56f7439a9e1fc82f153f018c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 07:59:12.248629 10751 log.cc:1196] T 2e540f389b2a4aef93a020f549d11b67 P 8315d22c56f7439a9e1fc82f153f018c: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/2e540f389b2a4aef93a020f549d11b67
I20240501 07:59:12.249745 11317 ts_tablet_manager.cc:902] T ae46990350284295bf14aa64d4b6c93e P a505e7cb133c4392bfda8a4a20874ae9: Initiating tablet copy from peer 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132:36773)
I20240501 07:59:12.249743 6159 catalog_manager.cc:4830] TS 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130:32941): tablet 2e540f389b2a4aef93a020f549d11b67 (table rebalance_test_table_15 [id=ecf5cb7235314f20bbb522737ba7fd3c]) successfully deleted
I20240501 07:59:12.250306 11317 tablet_copy_client.cc:308] T ae46990350284295bf14aa64d4b6c93e P a505e7cb133c4392bfda8a4a20874ae9: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:36773
I20240501 07:59:12.250813 6728 tablet_copy_service.cc:140] P 1f9cb72be98941b5a8e504cbe34132b9: Received BeginTabletCopySession request for tablet ae46990350284295bf14aa64d4b6c93e from peer a505e7cb133c4392bfda8a4a20874ae9 ({username='slave'} at 127.0.99.135:56663)
I20240501 07:59:12.250922 6728 tablet_copy_service.cc:161] P 1f9cb72be98941b5a8e504cbe34132b9: Beginning new tablet copy session on tablet ae46990350284295bf14aa64d4b6c93e from peer a505e7cb133c4392bfda8a4a20874ae9 at {username='slave'} at 127.0.99.135:56663: session id = a505e7cb133c4392bfda8a4a20874ae9-ae46990350284295bf14aa64d4b6c93e
I20240501 07:59:12.252022 6728 tablet_copy_source_session.cc:205] T ae46990350284295bf14aa64d4b6c93e P 1f9cb72be98941b5a8e504cbe34132b9: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:12.253296 11317 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae46990350284295bf14aa64d4b6c93e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:12.255796 11317 tablet_copy_client.cc:783] T ae46990350284295bf14aa64d4b6c93e P a505e7cb133c4392bfda8a4a20874ae9: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:12.255956 11317 tablet_copy_client.cc:647] T ae46990350284295bf14aa64d4b6c93e P a505e7cb133c4392bfda8a4a20874ae9: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:12.258855 11317 tablet_copy_client.cc:523] T ae46990350284295bf14aa64d4b6c93e P a505e7cb133c4392bfda8a4a20874ae9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:12.260270 11317 tablet_bootstrap.cc:492] T ae46990350284295bf14aa64d4b6c93e P a505e7cb133c4392bfda8a4a20874ae9: Bootstrap starting.
I20240501 07:59:12.264609 11339 ts_tablet_manager.cc:902] T 8454f24b0c28451b8ee51be39ca3e07f P 13d05ebdca8c4f489c597ce550dd773d: Initiating tablet copy from peer 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132:36773)
I20240501 07:59:12.268308 11339 tablet_copy_client.cc:308] T 8454f24b0c28451b8ee51be39ca3e07f P 13d05ebdca8c4f489c597ce550dd773d: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:36773
I20240501 07:59:12.269779 6728 tablet_copy_service.cc:140] P 1f9cb72be98941b5a8e504cbe34132b9: Received BeginTabletCopySession request for tablet 8454f24b0c28451b8ee51be39ca3e07f from peer 13d05ebdca8c4f489c597ce550dd773d ({username='slave'} at 127.0.99.134:44489)
I20240501 07:59:12.269904 6728 tablet_copy_service.cc:161] P 1f9cb72be98941b5a8e504cbe34132b9: Beginning new tablet copy session on tablet 8454f24b0c28451b8ee51be39ca3e07f from peer 13d05ebdca8c4f489c597ce550dd773d at {username='slave'} at 127.0.99.134:44489: session id = 13d05ebdca8c4f489c597ce550dd773d-8454f24b0c28451b8ee51be39ca3e07f
I20240501 07:59:12.270944 6728 tablet_copy_source_session.cc:205] T 8454f24b0c28451b8ee51be39ca3e07f P 1f9cb72be98941b5a8e504cbe34132b9: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:12.273558 11339 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8454f24b0c28451b8ee51be39ca3e07f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:12.277925 11339 tablet_copy_client.cc:783] T 8454f24b0c28451b8ee51be39ca3e07f P 13d05ebdca8c4f489c597ce550dd773d: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:12.278193 11339 tablet_copy_client.cc:647] T 8454f24b0c28451b8ee51be39ca3e07f P 13d05ebdca8c4f489c597ce550dd773d: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:12.279505 11339 tablet_copy_client.cc:523] T 8454f24b0c28451b8ee51be39ca3e07f P 13d05ebdca8c4f489c597ce550dd773d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:12.280990 11339 tablet_bootstrap.cc:492] T 8454f24b0c28451b8ee51be39ca3e07f P 13d05ebdca8c4f489c597ce550dd773d: Bootstrap starting.
I20240501 07:59:12.291862 6169 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55950:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 07:59:12.292032 6169 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55950:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 07:59:12.299669 6311 consensus_queue.cc:237] T 6c3c0d6fe41f4ae196ae29beb9b68125 P 8b8b3ad3aa8f4053b2167dd550f2086f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false } }
I20240501 07:59:12.300930 6577 raft_consensus.cc:1261] T 6c3c0d6fe41f4ae196ae29beb9b68125 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Refusing update from remote peer 8b8b3ad3aa8f4053b2167dd550f2086f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 07:59:12.300993 9056 raft_consensus.cc:1261] T 6c3c0d6fe41f4ae196ae29beb9b68125 P a505e7cb133c4392bfda8a4a20874ae9 [term 1 FOLLOWER]: Refusing update from remote peer 8b8b3ad3aa8f4053b2167dd550f2086f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 07:59:12.301432 11085 consensus_queue.cc:1035] T 6c3c0d6fe41f4ae196ae29beb9b68125 P 8b8b3ad3aa8f4053b2167dd550f2086f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 07:59:12.301612 9373 consensus_queue.cc:1035] T 6c3c0d6fe41f4ae196ae29beb9b68125 P 8b8b3ad3aa8f4053b2167dd550f2086f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 07:59:12.303167 6169 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 0bc706e85ce74cef97a4827ea4d4aa65: Sending DeleteTablet for 3 replicas of tablet 1eae52e7f97f45b48697c624192aa14b
I20240501 07:59:12.303737 9167 tablet_service.cc:1514] Processing DeleteTablet for tablet 1eae52e7f97f45b48697c624192aa14b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:12 UTC) from {username='slave'} at 127.0.0.1:33594
I20240501 07:59:12.303930 11153 tablet_replica.cc:330] T 1eae52e7f97f45b48697c624192aa14b P 83f20e84af4c40d09a4804b6b26e697f: stopping tablet replica
I20240501 07:59:12.304075 11153 raft_consensus.cc:2229] T 1eae52e7f97f45b48697c624192aa14b P 83f20e84af4c40d09a4804b6b26e697f [term 1 LEADER]: Raft consensus shutting down.
I20240501 07:59:12.304047 8774 tablet_service.cc:1514] Processing DeleteTablet for tablet 1eae52e7f97f45b48697c624192aa14b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:12 UTC) from {username='slave'} at 127.0.0.1:43136
I20240501 07:59:12.304261 11226 tablet_replica.cc:330] T 1eae52e7f97f45b48697c624192aa14b P 74077b61d6f54342bf9736b4a9f2eade: stopping tablet replica
I20240501 07:59:12.304302 11153 raft_consensus.cc:2258] T 1eae52e7f97f45b48697c624192aa14b P 83f20e84af4c40d09a4804b6b26e697f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:12.304395 11226 raft_consensus.cc:2229] T 1eae52e7f97f45b48697c624192aa14b P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:12.304488 6687 tablet_service.cc:1514] Processing DeleteTablet for tablet 1eae52e7f97f45b48697c624192aa14b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:12 UTC) from {username='slave'} at 127.0.0.1:38524
I20240501 07:59:12.304531 11226 raft_consensus.cc:2258] T 1eae52e7f97f45b48697c624192aa14b P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:12.304664 10310 tablet_replica.cc:330] T 1eae52e7f97f45b48697c624192aa14b P 1f9cb72be98941b5a8e504cbe34132b9: stopping tablet replica
I20240501 07:59:12.304821 10310 raft_consensus.cc:2229] T 1eae52e7f97f45b48697c624192aa14b P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:12.304949 10310 raft_consensus.cc:2258] T 1eae52e7f97f45b48697c624192aa14b P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:12.305408 11153 ts_tablet_manager.cc:1875] T 1eae52e7f97f45b48697c624192aa14b P 83f20e84af4c40d09a4804b6b26e697f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:59:12.305428 11226 ts_tablet_manager.cc:1875] T 1eae52e7f97f45b48697c624192aa14b P 74077b61d6f54342bf9736b4a9f2eade: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:59:12.305379 9373 raft_consensus.cc:2932] T 6c3c0d6fe41f4ae196ae29beb9b68125 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false } } }
I20240501 07:59:12.305836 10310 ts_tablet_manager.cc:1875] T 1eae52e7f97f45b48697c624192aa14b P 1f9cb72be98941b5a8e504cbe34132b9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:59:12.307421 6157 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6c3c0d6fe41f4ae196ae29beb9b68125 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 07:59:12.307579 11153 ts_tablet_manager.cc:1888] T 1eae52e7f97f45b48697c624192aa14b P 83f20e84af4c40d09a4804b6b26e697f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:59:12.307596 11226 ts_tablet_manager.cc:1888] T 1eae52e7f97f45b48697c624192aa14b P 74077b61d6f54342bf9736b4a9f2eade: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:59:12.307650 11226 log.cc:1196] T 1eae52e7f97f45b48697c624192aa14b P 74077b61d6f54342bf9736b4a9f2eade: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/1eae52e7f97f45b48697c624192aa14b
I20240501 07:59:12.307650 11153 log.cc:1196] T 1eae52e7f97f45b48697c624192aa14b P 83f20e84af4c40d09a4804b6b26e697f: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/1eae52e7f97f45b48697c624192aa14b
I20240501 07:59:12.307917 11153 ts_tablet_manager.cc:1909] T 1eae52e7f97f45b48697c624192aa14b P 83f20e84af4c40d09a4804b6b26e697f: Deleting consensus metadata
I20240501 07:59:12.308032 10310 ts_tablet_manager.cc:1888] T 1eae52e7f97f45b48697c624192aa14b P 1f9cb72be98941b5a8e504cbe34132b9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:59:12.308094 10310 log.cc:1196] T 1eae52e7f97f45b48697c624192aa14b P 1f9cb72be98941b5a8e504cbe34132b9: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/1eae52e7f97f45b48697c624192aa14b
I20240501 07:59:12.308336 10310 ts_tablet_manager.cc:1909] T 1eae52e7f97f45b48697c624192aa14b P 1f9cb72be98941b5a8e504cbe34132b9: Deleting consensus metadata
I20240501 07:59:12.308599 6158 catalog_manager.cc:4830] TS 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): tablet 1eae52e7f97f45b48697c624192aa14b (table rebalancer_dynamic_table_new_name [id=a57717a5df61497abe35bc282984a9ab]) successfully deleted
I20240501 07:59:12.309070 6577 raft_consensus.cc:2932] T 6c3c0d6fe41f4ae196ae29beb9b68125 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false } } }
I20240501 07:59:12.309235 9056 raft_consensus.cc:2932] T 6c3c0d6fe41f4ae196ae29beb9b68125 P a505e7cb133c4392bfda8a4a20874ae9 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false } } }
I20240501 07:59:12.309645 6169 catalog_manager.cc:5484] T 6c3c0d6fe41f4ae196ae29beb9b68125 P 8b8b3ad3aa8f4053b2167dd550f2086f reported cstate change: config changed from index 5 to 6, VOTER 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 07:59:12.311467 6159 catalog_manager.cc:4830] TS 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132:36773): tablet 1eae52e7f97f45b48697c624192aa14b (table rebalancer_dynamic_table_new_name [id=a57717a5df61497abe35bc282984a9ab]) successfully deleted
I20240501 07:59:12.312340 11226 ts_tablet_manager.cc:1909] T 1eae52e7f97f45b48697c624192aa14b P 74077b61d6f54342bf9736b4a9f2eade: Deleting consensus metadata
I20240501 07:59:12.313099 6687 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c3c0d6fe41f4ae196ae29beb9b68125 with delete_type TABLET_DATA_TOMBSTONED (TS 1f9cb72be98941b5a8e504cbe34132b9 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:38524
I20240501 07:59:12.313198 6157 catalog_manager.cc:4830] TS 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133:39859): tablet 1eae52e7f97f45b48697c624192aa14b (table rebalancer_dynamic_table_new_name [id=a57717a5df61497abe35bc282984a9ab]) successfully deleted
I20240501 07:59:12.313321 10310 tablet_replica.cc:330] T 6c3c0d6fe41f4ae196ae29beb9b68125 P 1f9cb72be98941b5a8e504cbe34132b9: stopping tablet replica
I20240501 07:59:12.313489 10310 raft_consensus.cc:2229] T 6c3c0d6fe41f4ae196ae29beb9b68125 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:12.313659 10310 raft_consensus.cc:2258] T 6c3c0d6fe41f4ae196ae29beb9b68125 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:12.315001 10310 ts_tablet_manager.cc:1875] T 6c3c0d6fe41f4ae196ae29beb9b68125 P 1f9cb72be98941b5a8e504cbe34132b9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:59:12.317167 11346 ts_tablet_manager.cc:902] T 7dd6566ccb6e43388146a6dfd1c4a88c P 13d05ebdca8c4f489c597ce550dd773d: Initiating tablet copy from peer 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132:36773)
I20240501 07:59:12.317727 11346 tablet_copy_client.cc:308] T 7dd6566ccb6e43388146a6dfd1c4a88c P 13d05ebdca8c4f489c597ce550dd773d: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:36773
I20240501 07:59:12.318426 6728 tablet_copy_service.cc:140] P 1f9cb72be98941b5a8e504cbe34132b9: Received BeginTabletCopySession request for tablet 7dd6566ccb6e43388146a6dfd1c4a88c from peer 13d05ebdca8c4f489c597ce550dd773d ({username='slave'} at 127.0.99.134:44489)
I20240501 07:59:12.318523 6728 tablet_copy_service.cc:161] P 1f9cb72be98941b5a8e504cbe34132b9: Beginning new tablet copy session on tablet 7dd6566ccb6e43388146a6dfd1c4a88c from peer 13d05ebdca8c4f489c597ce550dd773d at {username='slave'} at 127.0.99.134:44489: session id = 13d05ebdca8c4f489c597ce550dd773d-7dd6566ccb6e43388146a6dfd1c4a88c
I20240501 07:59:12.318508 9189 raft_consensus.cc:1203] T 782e37a4ebb34a68b461a74115ae8703 P 83f20e84af4c40d09a4804b6b26e697f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240501 07:59:12.319460 6728 tablet_copy_source_session.cc:205] T 7dd6566ccb6e43388146a6dfd1c4a88c P 1f9cb72be98941b5a8e504cbe34132b9: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:12.320335 10310 ts_tablet_manager.cc:1888] T 6c3c0d6fe41f4ae196ae29beb9b68125 P 1f9cb72be98941b5a8e504cbe34132b9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 07:59:12.320497 11346 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7dd6566ccb6e43388146a6dfd1c4a88c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:12.320998 10310 log.cc:1196] T 6c3c0d6fe41f4ae196ae29beb9b68125 P 1f9cb72be98941b5a8e504cbe34132b9: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/6c3c0d6fe41f4ae196ae29beb9b68125
I20240501 07:59:12.322352 6159 catalog_manager.cc:4830] TS 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132:36773): tablet 6c3c0d6fe41f4ae196ae29beb9b68125 (table rebalance_test_table_19 [id=35f40c7e6b1e4d799576a7d98130d848]) successfully deleted
I20240501 07:59:12.323745 11339 tablet_bootstrap.cc:492] T 8454f24b0c28451b8ee51be39ca3e07f P 13d05ebdca8c4f489c597ce550dd773d: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:12.323839 11346 tablet_copy_client.cc:783] T 7dd6566ccb6e43388146a6dfd1c4a88c P 13d05ebdca8c4f489c597ce550dd773d: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:12.323982 11346 tablet_copy_client.cc:647] T 7dd6566ccb6e43388146a6dfd1c4a88c P 13d05ebdca8c4f489c597ce550dd773d: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:12.324041 11339 tablet_bootstrap.cc:492] T 8454f24b0c28451b8ee51be39ca3e07f P 13d05ebdca8c4f489c597ce550dd773d: Bootstrap complete.
I20240501 07:59:12.324210 11339 ts_tablet_manager.cc:1367] T 8454f24b0c28451b8ee51be39ca3e07f P 13d05ebdca8c4f489c597ce550dd773d: Time spent bootstrapping tablet: real 0.043s user 0.027s sys 0.001s
I20240501 07:59:12.325400 11339 raft_consensus.cc:348] T 8454f24b0c28451b8ee51be39ca3e07f P 13d05ebdca8c4f489c597ce550dd773d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: true } }
I20240501 07:59:12.325531 11339 raft_consensus.cc:729] T 8454f24b0c28451b8ee51be39ca3e07f P 13d05ebdca8c4f489c597ce550dd773d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 13d05ebdca8c4f489c597ce550dd773d, State: Initialized, Role: LEARNER
I20240501 07:59:12.325639 11339 consensus_queue.cc:260] T 8454f24b0c28451b8ee51be39ca3e07f P 13d05ebdca8c4f489c597ce550dd773d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: true } }
I20240501 07:59:12.326618 11339 ts_tablet_manager.cc:1398] T 8454f24b0c28451b8ee51be39ca3e07f P 13d05ebdca8c4f489c597ce550dd773d: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240501 07:59:12.327030 6728 tablet_copy_service.cc:342] P 1f9cb72be98941b5a8e504cbe34132b9: Request end of tablet copy session 13d05ebdca8c4f489c597ce550dd773d-8454f24b0c28451b8ee51be39ca3e07f received from {username='slave'} at 127.0.99.134:44489
I20240501 07:59:12.327101 6728 tablet_copy_service.cc:433] P 1f9cb72be98941b5a8e504cbe34132b9: ending tablet copy session 13d05ebdca8c4f489c597ce550dd773d-8454f24b0c28451b8ee51be39ca3e07f on tablet 8454f24b0c28451b8ee51be39ca3e07f with peer 13d05ebdca8c4f489c597ce550dd773d
I20240501 07:59:12.334120 11346 tablet_copy_client.cc:523] T 7dd6566ccb6e43388146a6dfd1c4a88c P 13d05ebdca8c4f489c597ce550dd773d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:12.335830 11346 tablet_bootstrap.cc:492] T 7dd6566ccb6e43388146a6dfd1c4a88c P 13d05ebdca8c4f489c597ce550dd773d: Bootstrap starting.
I20240501 07:59:12.339320 8904 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 07:59:12.341264 6687 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 07:59:12.344854 11317 tablet_bootstrap.cc:492] T ae46990350284295bf14aa64d4b6c93e P a505e7cb133c4392bfda8a4a20874ae9: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:12.345207 11317 tablet_bootstrap.cc:492] T ae46990350284295bf14aa64d4b6c93e P a505e7cb133c4392bfda8a4a20874ae9: Bootstrap complete.
I20240501 07:59:12.345387 11317 ts_tablet_manager.cc:1367] T ae46990350284295bf14aa64d4b6c93e P a505e7cb133c4392bfda8a4a20874ae9: Time spent bootstrapping tablet: real 0.085s user 0.029s sys 0.000s
I20240501 07:59:12.345842 11317 raft_consensus.cc:348] T ae46990350284295bf14aa64d4b6c93e P a505e7cb133c4392bfda8a4a20874ae9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: true } }
I20240501 07:59:12.345975 11317 raft_consensus.cc:729] T ae46990350284295bf14aa64d4b6c93e P a505e7cb133c4392bfda8a4a20874ae9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a505e7cb133c4392bfda8a4a20874ae9, State: Initialized, Role: LEARNER
I20240501 07:59:12.346074 11317 consensus_queue.cc:260] T ae46990350284295bf14aa64d4b6c93e P a505e7cb133c4392bfda8a4a20874ae9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: true } }
I20240501 07:59:12.348058 11317 ts_tablet_manager.cc:1398] T ae46990350284295bf14aa64d4b6c93e P a505e7cb133c4392bfda8a4a20874ae9: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 07:59:12.348488 6728 tablet_copy_service.cc:342] P 1f9cb72be98941b5a8e504cbe34132b9: Request end of tablet copy session a505e7cb133c4392bfda8a4a20874ae9-ae46990350284295bf14aa64d4b6c93e received from {username='slave'} at 127.0.99.135:56663
I20240501 07:59:12.348567 6728 tablet_copy_service.cc:433] P 1f9cb72be98941b5a8e504cbe34132b9: ending tablet copy session a505e7cb133c4392bfda8a4a20874ae9-ae46990350284295bf14aa64d4b6c93e on tablet ae46990350284295bf14aa64d4b6c93e with peer a505e7cb133c4392bfda8a4a20874ae9
I20240501 07:59:12.352510 9167 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 07:59:12.354075 9400 consensus_queue.cc:1035] T 86e28c076f6749ceafe612bf730bfa68 P 13d05ebdca8c4f489c597ce550dd773d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:12.361532 9400 consensus_queue.cc:1035] T 86e28c076f6749ceafe612bf730bfa68 P 13d05ebdca8c4f489c597ce550dd773d [LEADER]: Connected to new peer: Peer: permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:12.368165 6294 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 07:59:12.369076 8774 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 07:59:12.370838 11346 tablet_bootstrap.cc:492] T 7dd6566ccb6e43388146a6dfd1c4a88c P 13d05ebdca8c4f489c597ce550dd773d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:12.371052 10945 consensus_queue.cc:1035] T 12ce6a21d06c4ba889542758be9dc62d P a505e7cb133c4392bfda8a4a20874ae9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:12.371156 11346 tablet_bootstrap.cc:492] T 7dd6566ccb6e43388146a6dfd1c4a88c P 13d05ebdca8c4f489c597ce550dd773d: Bootstrap complete.
I20240501 07:59:12.371326 11346 ts_tablet_manager.cc:1367] T 7dd6566ccb6e43388146a6dfd1c4a88c P 13d05ebdca8c4f489c597ce550dd773d: Time spent bootstrapping tablet: real 0.036s user 0.024s sys 0.005s
I20240501 07:59:12.371794 11346 raft_consensus.cc:348] T 7dd6566ccb6e43388146a6dfd1c4a88c P 13d05ebdca8c4f489c597ce550dd773d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: true } }
I20240501 07:59:12.371912 11346 raft_consensus.cc:729] T 7dd6566ccb6e43388146a6dfd1c4a88c P 13d05ebdca8c4f489c597ce550dd773d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 13d05ebdca8c4f489c597ce550dd773d, State: Initialized, Role: LEARNER
I20240501 07:59:12.372011 11346 consensus_queue.cc:260] T 7dd6566ccb6e43388146a6dfd1c4a88c P 13d05ebdca8c4f489c597ce550dd773d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: true } }
I20240501 07:59:12.372289 11346 ts_tablet_manager.cc:1398] T 7dd6566ccb6e43388146a6dfd1c4a88c P 13d05ebdca8c4f489c597ce550dd773d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:59:12.372697 6728 tablet_copy_service.cc:342] P 1f9cb72be98941b5a8e504cbe34132b9: Request end of tablet copy session 13d05ebdca8c4f489c597ce550dd773d-7dd6566ccb6e43388146a6dfd1c4a88c received from {username='slave'} at 127.0.99.134:44489
I20240501 07:59:12.372773 6728 tablet_copy_service.cc:433] P 1f9cb72be98941b5a8e504cbe34132b9: ending tablet copy session 13d05ebdca8c4f489c597ce550dd773d-7dd6566ccb6e43388146a6dfd1c4a88c on tablet 7dd6566ccb6e43388146a6dfd1c4a88c with peer 13d05ebdca8c4f489c597ce550dd773d
I20240501 07:59:12.378574 6171 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55950:
table { table_name: "rebalance_test_table_18" } modify_external_catalogs: true
I20240501 07:59:12.378727 6171 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55950:
table { table_name: "rebalance_test_table_18" } modify_external_catalogs: true
I20240501 07:59:12.381685 6171 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 0bc706e85ce74cef97a4827ea4d4aa65: Sending DeleteTablet for 4 replicas of tablet ae46990350284295bf14aa64d4b6c93e
I20240501 07:59:12.382022 6171 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 0bc706e85ce74cef97a4827ea4d4aa65: Sending DeleteTablet for 3 replicas of tablet b284ed2cb0264ced99a4ffd90983d902
I20240501 07:59:12.382288 6171 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 0bc706e85ce74cef97a4827ea4d4aa65: Sending DeleteTablet for 4 replicas of tablet aa1b8716228b4fd9af95870b7a613fb2
I20240501 07:59:12.382326 6687 tablet_service.cc:1514] Processing DeleteTablet for tablet ae46990350284295bf14aa64d4b6c93e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:12 UTC) from {username='slave'} at 127.0.0.1:38524
I20240501 07:59:12.382453 6556 tablet_service.cc:1514] Processing DeleteTablet for tablet b284ed2cb0264ced99a4ffd90983d902 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:12 UTC) from {username='slave'} at 127.0.0.1:53874
I20240501 07:59:12.382529 6424 tablet_service.cc:1514] Processing DeleteTablet for tablet ae46990350284295bf14aa64d4b6c93e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:12 UTC) from {username='slave'} at 127.0.0.1:45034
I20240501 07:59:12.382639 10310 tablet_replica.cc:330] T ae46990350284295bf14aa64d4b6c93e P 1f9cb72be98941b5a8e504cbe34132b9: stopping tablet replica
I20240501 07:59:12.382691 10751 tablet_replica.cc:330] T ae46990350284295bf14aa64d4b6c93e P 8315d22c56f7439a9e1fc82f153f018c: stopping tablet replica
I20240501 07:59:12.382814 10310 raft_consensus.cc:2229] T ae46990350284295bf14aa64d4b6c93e P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 LEADER]: Raft consensus shutting down.
I20240501 07:59:12.382835 10751 raft_consensus.cc:2229] T ae46990350284295bf14aa64d4b6c93e P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:12.382982 10751 raft_consensus.cc:2258] T ae46990350284295bf14aa64d4b6c93e P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:12.383105 10310 raft_consensus.cc:2258] T ae46990350284295bf14aa64d4b6c93e P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:12.383167 9035 tablet_service.cc:1514] Processing DeleteTablet for tablet ae46990350284295bf14aa64d4b6c93e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:12 UTC) from {username='slave'} at 127.0.0.1:44236
I20240501 07:59:12.383278 10753 tablet_replica.cc:330] T b284ed2cb0264ced99a4ffd90983d902 P 3b164cbbc00f4e8eb3254ddd349a3ff9: stopping tablet replica
I20240501 07:59:12.383359 10940 tablet_replica.cc:330] T ae46990350284295bf14aa64d4b6c93e P a505e7cb133c4392bfda8a4a20874ae9: stopping tablet replica
I20240501 07:59:12.383425 10753 raft_consensus.cc:2229] T b284ed2cb0264ced99a4ffd90983d902 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:12.383466 10940 raft_consensus.cc:2229] T ae46990350284295bf14aa64d4b6c93e P a505e7cb133c4392bfda8a4a20874ae9 [term 1 LEARNER]: Raft consensus shutting down.
I20240501 07:59:12.383553 10940 raft_consensus.cc:2258] T ae46990350284295bf14aa64d4b6c93e P a505e7cb133c4392bfda8a4a20874ae9 [term 1 LEARNER]: Raft consensus is shut down!
I20240501 07:59:12.383623 10753 raft_consensus.cc:2258] T b284ed2cb0264ced99a4ffd90983d902 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:12.383600 9167 tablet_service.cc:1514] Processing DeleteTablet for tablet aa1b8716228b4fd9af95870b7a613fb2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:12 UTC) from {username='slave'} at 127.0.0.1:33594
I20240501 07:59:12.383802 6686 tablet_service.cc:1514] Processing DeleteTablet for tablet aa1b8716228b4fd9af95870b7a613fb2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:12 UTC) from {username='slave'} at 127.0.0.1:38524
I20240501 07:59:12.383802 6424 tablet_service.cc:1514] Processing DeleteTablet for tablet aa1b8716228b4fd9af95870b7a613fb2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:12 UTC) from {username='slave'} at 127.0.0.1:45034
I20240501 07:59:12.383796 6294 tablet_service.cc:1514] Processing DeleteTablet for tablet ae46990350284295bf14aa64d4b6c93e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:12 UTC) from {username='slave'} at 127.0.0.1:45480
I20240501 07:59:12.383929 6687 tablet_service.cc:1514] Processing DeleteTablet for tablet b284ed2cb0264ced99a4ffd90983d902 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:12 UTC) from {username='slave'} at 127.0.0.1:38524
I20240501 07:59:12.383983 11153 tablet_replica.cc:330] T aa1b8716228b4fd9af95870b7a613fb2 P 83f20e84af4c40d09a4804b6b26e697f: stopping tablet replica
I20240501 07:59:12.384107 6295 tablet_service.cc:1514] Processing DeleteTablet for tablet aa1b8716228b4fd9af95870b7a613fb2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:12 UTC) from {username='slave'} at 127.0.0.1:45480
I20240501 07:59:12.384121 11153 raft_consensus.cc:2229] T aa1b8716228b4fd9af95870b7a613fb2 P 83f20e84af4c40d09a4804b6b26e697f [term 1 LEARNER]: Raft consensus shutting down.
I20240501 07:59:12.384241 11153 raft_consensus.cc:2258] T aa1b8716228b4fd9af95870b7a613fb2 P 83f20e84af4c40d09a4804b6b26e697f [term 1 LEARNER]: Raft consensus is shut down!
I20240501 07:59:12.384275 10309 tablet_replica.cc:330] T ae46990350284295bf14aa64d4b6c93e P 8b8b3ad3aa8f4053b2167dd550f2086f: stopping tablet replica
I20240501 07:59:12.384399 10309 raft_consensus.cc:2229] T ae46990350284295bf14aa64d4b6c93e P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:12.384479 10940 ts_tablet_manager.cc:1875] T ae46990350284295bf14aa64d4b6c93e P a505e7cb133c4392bfda8a4a20874ae9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:59:12.384546 10309 raft_consensus.cc:2258] T ae46990350284295bf14aa64d4b6c93e P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:12.384766 11153 ts_tablet_manager.cc:1875] T aa1b8716228b4fd9af95870b7a613fb2 P 83f20e84af4c40d09a4804b6b26e697f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:59:12.385982 10309 ts_tablet_manager.cc:1875] T ae46990350284295bf14aa64d4b6c93e P 8b8b3ad3aa8f4053b2167dd550f2086f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:59:12.386840 10940 ts_tablet_manager.cc:1888] T ae46990350284295bf14aa64d4b6c93e P a505e7cb133c4392bfda8a4a20874ae9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:59:12.386904 10940 log.cc:1196] T ae46990350284295bf14aa64d4b6c93e P a505e7cb133c4392bfda8a4a20874ae9: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/ae46990350284295bf14aa64d4b6c93e
I20240501 07:59:12.386955 11153 ts_tablet_manager.cc:1888] T aa1b8716228b4fd9af95870b7a613fb2 P 83f20e84af4c40d09a4804b6b26e697f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:59:12.387027 11153 log.cc:1196] T aa1b8716228b4fd9af95870b7a613fb2 P 83f20e84af4c40d09a4804b6b26e697f: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/aa1b8716228b4fd9af95870b7a613fb2
I20240501 07:59:12.387210 10940 ts_tablet_manager.cc:1909] T ae46990350284295bf14aa64d4b6c93e P a505e7cb133c4392bfda8a4a20874ae9: Deleting consensus metadata
I20240501 07:59:12.387307 11153 ts_tablet_manager.cc:1909] T aa1b8716228b4fd9af95870b7a613fb2 P 83f20e84af4c40d09a4804b6b26e697f: Deleting consensus metadata
I20240501 07:59:12.388247 10309 ts_tablet_manager.cc:1888] T ae46990350284295bf14aa64d4b6c93e P 8b8b3ad3aa8f4053b2167dd550f2086f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:59:12.388312 10309 log.cc:1196] T ae46990350284295bf14aa64d4b6c93e P 8b8b3ad3aa8f4053b2167dd550f2086f: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/ae46990350284295bf14aa64d4b6c93e
I20240501 07:59:12.388525 10309 ts_tablet_manager.cc:1909] T ae46990350284295bf14aa64d4b6c93e P 8b8b3ad3aa8f4053b2167dd550f2086f: Deleting consensus metadata
I20240501 07:59:12.388859 10751 ts_tablet_manager.cc:1875] T ae46990350284295bf14aa64d4b6c93e P 8315d22c56f7439a9e1fc82f153f018c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:59:12.389380 10753 ts_tablet_manager.cc:1875] T b284ed2cb0264ced99a4ffd90983d902 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:59:12.390897 10751 ts_tablet_manager.cc:1888] T ae46990350284295bf14aa64d4b6c93e P 8315d22c56f7439a9e1fc82f153f018c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:59:12.390957 10751 log.cc:1196] T ae46990350284295bf14aa64d4b6c93e P 8315d22c56f7439a9e1fc82f153f018c: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/ae46990350284295bf14aa64d4b6c93e
I20240501 07:59:12.390933 10945 consensus_queue.cc:1035] T 12ce6a21d06c4ba889542758be9dc62d P a505e7cb133c4392bfda8a4a20874ae9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:12.391206 10751 ts_tablet_manager.cc:1909] T ae46990350284295bf14aa64d4b6c93e P 8315d22c56f7439a9e1fc82f153f018c: Deleting consensus metadata
I20240501 07:59:12.391636 10753 ts_tablet_manager.cc:1888] T b284ed2cb0264ced99a4ffd90983d902 P 3b164cbbc00f4e8eb3254ddd349a3ff9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.6
I20240501 07:59:12.391698 10753 log.cc:1196] T b284ed2cb0264ced99a4ffd90983d902 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/b284ed2cb0264ced99a4ffd90983d902
I20240501 07:59:12.391996 10753 ts_tablet_manager.cc:1909] T b284ed2cb0264ced99a4ffd90983d902 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Deleting consensus metadata
I20240501 07:59:12.392783 6159 catalog_manager.cc:4830] TS 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130:32941): tablet ae46990350284295bf14aa64d4b6c93e (table rebalance_test_table_18 [id=25ecaaa6eafe48ecb3f3ff353a6d1170]) successfully deleted
I20240501 07:59:12.393040 10751 tablet_replica.cc:330] T aa1b8716228b4fd9af95870b7a613fb2 P 8315d22c56f7439a9e1fc82f153f018c: stopping tablet replica
I20240501 07:59:12.393190 10751 raft_consensus.cc:2229] T aa1b8716228b4fd9af95870b7a613fb2 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:12.393312 10310 ts_tablet_manager.cc:1875] T ae46990350284295bf14aa64d4b6c93e P 1f9cb72be98941b5a8e504cbe34132b9: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240501 07:59:12.393895 6644 consensus_peers.cc:485] T b284ed2cb0264ced99a4ffd90983d902 P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131:34471): Couldn't send request to peer 3b164cbbc00f4e8eb3254ddd349a3ff9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b284ed2cb0264ced99a4ffd90983d902. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:12.393337 10751 raft_consensus.cc:2258] T aa1b8716228b4fd9af95870b7a613fb2 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:12.394671 10751 ts_tablet_manager.cc:1875] T aa1b8716228b4fd9af95870b7a613fb2 P 8315d22c56f7439a9e1fc82f153f018c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:59:12.395604 10310 ts_tablet_manager.cc:1888] T ae46990350284295bf14aa64d4b6c93e P 1f9cb72be98941b5a8e504cbe34132b9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:59:12.395682 10310 log.cc:1196] T ae46990350284295bf14aa64d4b6c93e P 1f9cb72be98941b5a8e504cbe34132b9: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/ae46990350284295bf14aa64d4b6c93e
I20240501 07:59:12.395903 10310 ts_tablet_manager.cc:1909] T ae46990350284295bf14aa64d4b6c93e P 1f9cb72be98941b5a8e504cbe34132b9: Deleting consensus metadata
I20240501 07:59:12.396102 6160 catalog_manager.cc:4830] TS 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131:34471): tablet b284ed2cb0264ced99a4ffd90983d902 (table rebalance_test_table_18 [id=25ecaaa6eafe48ecb3f3ff353a6d1170]) successfully deleted
I20240501 07:59:12.396589 10310 tablet_replica.cc:330] T aa1b8716228b4fd9af95870b7a613fb2 P 1f9cb72be98941b5a8e504cbe34132b9: stopping tablet replica
I20240501 07:59:12.396713 10310 raft_consensus.cc:2229] T aa1b8716228b4fd9af95870b7a613fb2 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:12.396838 10310 raft_consensus.cc:2258] T aa1b8716228b4fd9af95870b7a613fb2 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:12.397270 6159 catalog_manager.cc:4830] TS 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132:36773): tablet ae46990350284295bf14aa64d4b6c93e (table rebalance_test_table_18 [id=25ecaaa6eafe48ecb3f3ff353a6d1170]) successfully deleted
I20240501 07:59:12.397363 10310 ts_tablet_manager.cc:1875] T aa1b8716228b4fd9af95870b7a613fb2 P 1f9cb72be98941b5a8e504cbe34132b9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:59:12.397456 10309 tablet_replica.cc:330] T aa1b8716228b4fd9af95870b7a613fb2 P 8b8b3ad3aa8f4053b2167dd550f2086f: stopping tablet replica
I20240501 07:59:12.397585 10309 raft_consensus.cc:2229] T aa1b8716228b4fd9af95870b7a613fb2 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 LEADER]: Raft consensus shutting down.
I20240501 07:59:12.397760 8904 tablet_service.cc:1514] Processing DeleteTablet for tablet b284ed2cb0264ced99a4ffd90983d902 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:12 UTC) from {username='slave'} at 127.0.0.1:47202
I20240501 07:59:12.397820 10309 raft_consensus.cc:2258] T aa1b8716228b4fd9af95870b7a613fb2 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:12.398094 6157 catalog_manager.cc:4830] TS 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129:46875): tablet ae46990350284295bf14aa64d4b6c93e (table rebalance_test_table_18 [id=25ecaaa6eafe48ecb3f3ff353a6d1170]) successfully deleted
I20240501 07:59:12.398352 11370 tablet_replica.cc:330] T b284ed2cb0264ced99a4ffd90983d902 P 13d05ebdca8c4f489c597ce550dd773d: stopping tablet replica
I20240501 07:59:12.398509 11370 raft_consensus.cc:2229] T b284ed2cb0264ced99a4ffd90983d902 P 13d05ebdca8c4f489c597ce550dd773d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:12.398663 11370 raft_consensus.cc:2258] T b284ed2cb0264ced99a4ffd90983d902 P 13d05ebdca8c4f489c597ce550dd773d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:12.398787 10309 ts_tablet_manager.cc:1875] T aa1b8716228b4fd9af95870b7a613fb2 P 8b8b3ad3aa8f4053b2167dd550f2086f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:59:12.399282 11370 ts_tablet_manager.cc:1875] T b284ed2cb0264ced99a4ffd90983d902 P 13d05ebdca8c4f489c597ce550dd773d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:59:12.400126 10310 ts_tablet_manager.cc:1888] T aa1b8716228b4fd9af95870b7a613fb2 P 1f9cb72be98941b5a8e504cbe34132b9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:59:12.400206 10310 log.cc:1196] T aa1b8716228b4fd9af95870b7a613fb2 P 1f9cb72be98941b5a8e504cbe34132b9: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/aa1b8716228b4fd9af95870b7a613fb2
I20240501 07:59:12.400493 10310 ts_tablet_manager.cc:1909] T aa1b8716228b4fd9af95870b7a613fb2 P 1f9cb72be98941b5a8e504cbe34132b9: Deleting consensus metadata
I20240501 07:59:12.401067 10310 tablet_replica.cc:330] T b284ed2cb0264ced99a4ffd90983d902 P 1f9cb72be98941b5a8e504cbe34132b9: stopping tablet replica
I20240501 07:59:12.401206 10310 raft_consensus.cc:2229] T b284ed2cb0264ced99a4ffd90983d902 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 LEADER]: Raft consensus shutting down.
I20240501 07:59:12.401392 6159 catalog_manager.cc:4830] TS 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132:36773): tablet aa1b8716228b4fd9af95870b7a613fb2 (table rebalance_test_table_18 [id=25ecaaa6eafe48ecb3f3ff353a6d1170]) successfully deleted
I20240501 07:59:12.401422 10310 raft_consensus.cc:2258] T b284ed2cb0264ced99a4ffd90983d902 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:12.401973 10310 ts_tablet_manager.cc:1875] T b284ed2cb0264ced99a4ffd90983d902 P 1f9cb72be98941b5a8e504cbe34132b9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:59:12.402879 6158 catalog_manager.cc:4830] TS 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): tablet aa1b8716228b4fd9af95870b7a613fb2 (table rebalance_test_table_18 [id=25ecaaa6eafe48ecb3f3ff353a6d1170]) successfully deleted
I20240501 07:59:12.403007 6158 catalog_manager.cc:4830] TS a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): tablet ae46990350284295bf14aa64d4b6c93e (table rebalance_test_table_18 [id=25ecaaa6eafe48ecb3f3ff353a6d1170]) successfully deleted
I20240501 07:59:12.403923 11370 ts_tablet_manager.cc:1888] T b284ed2cb0264ced99a4ffd90983d902 P 13d05ebdca8c4f489c597ce550dd773d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.6
I20240501 07:59:12.404182 6424 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 07:59:12.404029 11370 log.cc:1196] T b284ed2cb0264ced99a4ffd90983d902 P 13d05ebdca8c4f489c597ce550dd773d: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/b284ed2cb0264ced99a4ffd90983d902
I20240501 07:59:12.404804 6171 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55950:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 07:59:12.405171 10310 ts_tablet_manager.cc:1888] T b284ed2cb0264ced99a4ffd90983d902 P 1f9cb72be98941b5a8e504cbe34132b9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.6
I20240501 07:59:12.405244 10310 log.cc:1196] T b284ed2cb0264ced99a4ffd90983d902 P 1f9cb72be98941b5a8e504cbe34132b9: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/b284ed2cb0264ced99a4ffd90983d902
I20240501 07:59:12.405329 9035 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 07:59:12.405508 10310 ts_tablet_manager.cc:1909] T b284ed2cb0264ced99a4ffd90983d902 P 1f9cb72be98941b5a8e504cbe34132b9: Deleting consensus metadata
I20240501 07:59:12.405792 11370 ts_tablet_manager.cc:1909] T b284ed2cb0264ced99a4ffd90983d902 P 13d05ebdca8c4f489c597ce550dd773d: Deleting consensus metadata
I20240501 07:59:12.406873 6159 catalog_manager.cc:4830] TS 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134:40141): tablet b284ed2cb0264ced99a4ffd90983d902 (table rebalance_test_table_18 [id=25ecaaa6eafe48ecb3f3ff353a6d1170]) successfully deleted
I20240501 07:59:12.407016 6159 catalog_manager.cc:4830] TS 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132:36773): tablet b284ed2cb0264ced99a4ffd90983d902 (table rebalance_test_table_18 [id=25ecaaa6eafe48ecb3f3ff353a6d1170]) successfully deleted
I20240501 07:59:12.413913 6556 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 07:59:12.414523 10309 ts_tablet_manager.cc:1888] T aa1b8716228b4fd9af95870b7a613fb2 P 8b8b3ad3aa8f4053b2167dd550f2086f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:59:12.414626 10309 log.cc:1196] T aa1b8716228b4fd9af95870b7a613fb2 P 8b8b3ad3aa8f4053b2167dd550f2086f: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/aa1b8716228b4fd9af95870b7a613fb2
I20240501 07:59:12.417083 10309 ts_tablet_manager.cc:1909] T aa1b8716228b4fd9af95870b7a613fb2 P 8b8b3ad3aa8f4053b2167dd550f2086f: Deleting consensus metadata
I20240501 07:59:12.417896 9035 tablet_service.cc:1467] Processing CreateTablet for tablet 32ecf214d6e14fa3be1b8eafd8c23fac (DEFAULT_TABLE table=rebalancer_dynamic_table [id=95208beb4f5944b09e76c37f268c35ba]), partition=
I20240501 07:59:12.418167 9035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32ecf214d6e14fa3be1b8eafd8c23fac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:12.419243 8793 raft_consensus.cc:1203] T 6e91683ff22243c1b3877733d74e2886 P 74077b61d6f54342bf9736b4a9f2eade [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 07:59:12.421455 6157 catalog_manager.cc:4830] TS 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129:46875): tablet aa1b8716228b4fd9af95870b7a613fb2 (table rebalance_test_table_18 [id=25ecaaa6eafe48ecb3f3ff353a6d1170]) successfully deleted
I20240501 07:59:12.422283 8774 tablet_service.cc:1467] Processing CreateTablet for tablet 32ecf214d6e14fa3be1b8eafd8c23fac (DEFAULT_TABLE table=rebalancer_dynamic_table [id=95208beb4f5944b09e76c37f268c35ba]), partition=
I20240501 07:59:12.422509 8774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32ecf214d6e14fa3be1b8eafd8c23fac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:12.424430 11372 tablet_bootstrap.cc:492] T 32ecf214d6e14fa3be1b8eafd8c23fac P 74077b61d6f54342bf9736b4a9f2eade: Bootstrap starting.
I20240501 07:59:12.425293 11372 tablet_bootstrap.cc:654] T 32ecf214d6e14fa3be1b8eafd8c23fac P 74077b61d6f54342bf9736b4a9f2eade: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:12.431361 8904 tablet_service.cc:1467] Processing CreateTablet for tablet 32ecf214d6e14fa3be1b8eafd8c23fac (DEFAULT_TABLE table=rebalancer_dynamic_table [id=95208beb4f5944b09e76c37f268c35ba]), partition=
I20240501 07:59:12.431631 8904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32ecf214d6e14fa3be1b8eafd8c23fac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:12.437425 10751 ts_tablet_manager.cc:1888] T aa1b8716228b4fd9af95870b7a613fb2 P 8315d22c56f7439a9e1fc82f153f018c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:59:12.437579 10751 log.cc:1196] T aa1b8716228b4fd9af95870b7a613fb2 P 8315d22c56f7439a9e1fc82f153f018c: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/aa1b8716228b4fd9af95870b7a613fb2
I20240501 07:59:12.437916 10751 ts_tablet_manager.cc:1909] T aa1b8716228b4fd9af95870b7a613fb2 P 8315d22c56f7439a9e1fc82f153f018c: Deleting consensus metadata
I20240501 07:59:12.439344 6159 catalog_manager.cc:4830] TS 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130:32941): tablet aa1b8716228b4fd9af95870b7a613fb2 (table rebalance_test_table_18 [id=25ecaaa6eafe48ecb3f3ff353a6d1170]) successfully deleted
I20240501 07:59:12.442409 11374 tablet_bootstrap.cc:492] T 32ecf214d6e14fa3be1b8eafd8c23fac P 13d05ebdca8c4f489c597ce550dd773d: Bootstrap starting.
I20240501 07:59:12.440928 11371 tablet_bootstrap.cc:492] T 32ecf214d6e14fa3be1b8eafd8c23fac P a505e7cb133c4392bfda8a4a20874ae9: Bootstrap starting.
I20240501 07:59:12.443852 11371 tablet_bootstrap.cc:654] T 32ecf214d6e14fa3be1b8eafd8c23fac P a505e7cb133c4392bfda8a4a20874ae9: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:12.444700 11372 tablet_bootstrap.cc:492] T 32ecf214d6e14fa3be1b8eafd8c23fac P 74077b61d6f54342bf9736b4a9f2eade: No bootstrap required, opened a new log
I20240501 07:59:12.444794 11372 ts_tablet_manager.cc:1367] T 32ecf214d6e14fa3be1b8eafd8c23fac P 74077b61d6f54342bf9736b4a9f2eade: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.000s
I20240501 07:59:12.445276 11372 raft_consensus.cc:348] T 32ecf214d6e14fa3be1b8eafd8c23fac P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } }
I20240501 07:59:12.445430 11372 raft_consensus.cc:374] T 32ecf214d6e14fa3be1b8eafd8c23fac P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:12.445473 11372 raft_consensus.cc:729] T 32ecf214d6e14fa3be1b8eafd8c23fac P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74077b61d6f54342bf9736b4a9f2eade, State: Initialized, Role: FOLLOWER
I20240501 07:59:12.445606 11372 consensus_queue.cc:260] T 32ecf214d6e14fa3be1b8eafd8c23fac P 74077b61d6f54342bf9736b4a9f2eade [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } }
I20240501 07:59:12.445899 11372 ts_tablet_manager.cc:1398] T 32ecf214d6e14fa3be1b8eafd8c23fac P 74077b61d6f54342bf9736b4a9f2eade: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:12.449424 11374 tablet_bootstrap.cc:654] T 32ecf214d6e14fa3be1b8eafd8c23fac P 13d05ebdca8c4f489c597ce550dd773d: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:12.453738 11371 tablet_bootstrap.cc:492] T 32ecf214d6e14fa3be1b8eafd8c23fac P a505e7cb133c4392bfda8a4a20874ae9: No bootstrap required, opened a new log
I20240501 07:59:12.453918 11371 ts_tablet_manager.cc:1367] T 32ecf214d6e14fa3be1b8eafd8c23fac P a505e7cb133c4392bfda8a4a20874ae9: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20240501 07:59:12.454638 11371 raft_consensus.cc:348] T 32ecf214d6e14fa3be1b8eafd8c23fac P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } }
I20240501 07:59:12.454751 11371 raft_consensus.cc:374] T 32ecf214d6e14fa3be1b8eafd8c23fac P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:12.454784 11371 raft_consensus.cc:729] T 32ecf214d6e14fa3be1b8eafd8c23fac P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a505e7cb133c4392bfda8a4a20874ae9, State: Initialized, Role: FOLLOWER
I20240501 07:59:12.454929 11371 consensus_queue.cc:260] T 32ecf214d6e14fa3be1b8eafd8c23fac P a505e7cb133c4392bfda8a4a20874ae9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } }
I20240501 07:59:12.455215 11371 ts_tablet_manager.cc:1398] T 32ecf214d6e14fa3be1b8eafd8c23fac P a505e7cb133c4392bfda8a4a20874ae9: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:59:12.456162 11374 tablet_bootstrap.cc:492] T 32ecf214d6e14fa3be1b8eafd8c23fac P 13d05ebdca8c4f489c597ce550dd773d: No bootstrap required, opened a new log
I20240501 07:59:12.456331 11374 ts_tablet_manager.cc:1367] T 32ecf214d6e14fa3be1b8eafd8c23fac P 13d05ebdca8c4f489c597ce550dd773d: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.000s
I20240501 07:59:12.458104 11374 raft_consensus.cc:348] T 32ecf214d6e14fa3be1b8eafd8c23fac P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } }
I20240501 07:59:12.458235 11374 raft_consensus.cc:374] T 32ecf214d6e14fa3be1b8eafd8c23fac P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:12.458293 11374 raft_consensus.cc:729] T 32ecf214d6e14fa3be1b8eafd8c23fac P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13d05ebdca8c4f489c597ce550dd773d, State: Initialized, Role: FOLLOWER
I20240501 07:59:12.458442 11374 consensus_queue.cc:260] T 32ecf214d6e14fa3be1b8eafd8c23fac P 13d05ebdca8c4f489c597ce550dd773d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } }
I20240501 07:59:12.458788 11374 ts_tablet_manager.cc:1398] T 32ecf214d6e14fa3be1b8eafd8c23fac P 13d05ebdca8c4f489c597ce550dd773d: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 07:59:12.487881 6169 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55950:
name: "rebalancer_extra_table_12"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 07:59:12.490566 8924 consensus_queue.cc:237] T 239ee596e62d47169eeb6670dc62d59c P 13d05ebdca8c4f489c597ce550dd773d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { promote: true } }
I20240501 07:59:12.492516 9056 raft_consensus.cc:1261] T 239ee596e62d47169eeb6670dc62d59c P a505e7cb133c4392bfda8a4a20874ae9 [term 1 FOLLOWER]: Refusing update from remote peer 13d05ebdca8c4f489c597ce550dd773d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 07:59:12.492717 8860 consensus_peers.cc:485] T 239ee596e62d47169eeb6670dc62d59c P 13d05ebdca8c4f489c597ce550dd773d -> Peer 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132:36773): Couldn't send request to peer 1f9cb72be98941b5a8e504cbe34132b9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 239ee596e62d47169eeb6670dc62d59c. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:12.493046 10235 consensus_queue.cc:1035] T 239ee596e62d47169eeb6670dc62d59c P 13d05ebdca8c4f489c597ce550dd773d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 07:59:12.493342 8794 raft_consensus.cc:1261] T 239ee596e62d47169eeb6670dc62d59c P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Refusing update from remote peer 13d05ebdca8c4f489c597ce550dd773d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:12.493883 10235 consensus_queue.cc:1035] T 239ee596e62d47169eeb6670dc62d59c P 13d05ebdca8c4f489c597ce550dd773d [LEADER]: Connected to new peer: Peer: permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 07:59:12.495761 8904 tablet_service.cc:1467] Processing CreateTablet for tablet 0864c290fafa488abefa93d41c0a2efe (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=731fe27ae8e34d92bd595d43c4bc8f3a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:12.495918 9169 tablet_service.cc:1467] Processing CreateTablet for tablet 59e171892fc1414fbcfd6a8c090854f3 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=731fe27ae8e34d92bd595d43c4bc8f3a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:12.496071 8904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0864c290fafa488abefa93d41c0a2efe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:12.496222 9169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59e171892fc1414fbcfd6a8c090854f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:12.496281 9167 tablet_service.cc:1467] Processing CreateTablet for tablet 0864c290fafa488abefa93d41c0a2efe (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=731fe27ae8e34d92bd595d43c4bc8f3a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:12.496529 9167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0864c290fafa488abefa93d41c0a2efe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:12.497474 9035 tablet_service.cc:1467] Processing CreateTablet for tablet 6daebcfc41754ec4aca939ca20588854 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=731fe27ae8e34d92bd595d43c4bc8f3a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:12.497769 9035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6daebcfc41754ec4aca939ca20588854. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:12.498569 8774 tablet_service.cc:1467] Processing CreateTablet for tablet 0864c290fafa488abefa93d41c0a2efe (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=731fe27ae8e34d92bd595d43c4bc8f3a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:12.498809 8774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0864c290fafa488abefa93d41c0a2efe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:12.498934 11385 tablet_bootstrap.cc:492] T 59e171892fc1414fbcfd6a8c090854f3 P 83f20e84af4c40d09a4804b6b26e697f: Bootstrap starting.
I20240501 07:59:12.499380 8757 tablet_service.cc:1467] Processing CreateTablet for tablet 59e171892fc1414fbcfd6a8c090854f3 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=731fe27ae8e34d92bd595d43c4bc8f3a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:12.499626 8757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59e171892fc1414fbcfd6a8c090854f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:12.499987 11371 tablet_bootstrap.cc:492] T 6daebcfc41754ec4aca939ca20588854 P a505e7cb133c4392bfda8a4a20874ae9: Bootstrap starting.
I20240501 07:59:12.500809 11374 tablet_bootstrap.cc:492] T 0864c290fafa488abefa93d41c0a2efe P 13d05ebdca8c4f489c597ce550dd773d: Bootstrap starting.
I20240501 07:59:12.500947 11371 tablet_bootstrap.cc:654] T 6daebcfc41754ec4aca939ca20588854 P a505e7cb133c4392bfda8a4a20874ae9: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:12.501266 10235 raft_consensus.cc:2932] T 239ee596e62d47169eeb6670dc62d59c P 13d05ebdca8c4f489c597ce550dd773d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { promote: true } } }
I20240501 07:59:12.501827 11372 tablet_bootstrap.cc:492] T 0864c290fafa488abefa93d41c0a2efe P 74077b61d6f54342bf9736b4a9f2eade: Bootstrap starting.
I20240501 07:59:12.501869 11374 tablet_bootstrap.cc:654] T 0864c290fafa488abefa93d41c0a2efe P 13d05ebdca8c4f489c597ce550dd773d: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:12.502832 11372 tablet_bootstrap.cc:654] T 0864c290fafa488abefa93d41c0a2efe P 74077b61d6f54342bf9736b4a9f2eade: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:12.503880 6687 tablet_service.cc:1467] Processing CreateTablet for tablet 6daebcfc41754ec4aca939ca20588854 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=731fe27ae8e34d92bd595d43c4bc8f3a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:12.504150 11374 tablet_bootstrap.cc:492] T 0864c290fafa488abefa93d41c0a2efe P 13d05ebdca8c4f489c597ce550dd773d: No bootstrap required, opened a new log
I20240501 07:59:12.504222 6687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6daebcfc41754ec4aca939ca20588854. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:12.504232 11374 ts_tablet_manager.cc:1367] T 0864c290fafa488abefa93d41c0a2efe P 13d05ebdca8c4f489c597ce550dd773d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 07:59:12.504743 11374 raft_consensus.cc:348] T 0864c290fafa488abefa93d41c0a2efe P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } }
I20240501 07:59:12.504848 11374 raft_consensus.cc:374] T 0864c290fafa488abefa93d41c0a2efe P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:12.504879 11374 raft_consensus.cc:729] T 0864c290fafa488abefa93d41c0a2efe P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13d05ebdca8c4f489c597ce550dd773d, State: Initialized, Role: FOLLOWER
I20240501 07:59:12.504951 11372 tablet_bootstrap.cc:492] T 0864c290fafa488abefa93d41c0a2efe P 74077b61d6f54342bf9736b4a9f2eade: No bootstrap required, opened a new log
I20240501 07:59:12.505059 11372 ts_tablet_manager.cc:1367] T 0864c290fafa488abefa93d41c0a2efe P 74077b61d6f54342bf9736b4a9f2eade: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 07:59:12.505028 11374 consensus_queue.cc:260] T 0864c290fafa488abefa93d41c0a2efe P 13d05ebdca8c4f489c597ce550dd773d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } }
I20240501 07:59:12.505323 11374 ts_tablet_manager.cc:1398] T 0864c290fafa488abefa93d41c0a2efe P 13d05ebdca8c4f489c597ce550dd773d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:12.505502 11372 raft_consensus.cc:348] T 0864c290fafa488abefa93d41c0a2efe P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } }
I20240501 07:59:12.505610 11372 raft_consensus.cc:374] T 0864c290fafa488abefa93d41c0a2efe P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:12.505647 11372 raft_consensus.cc:729] T 0864c290fafa488abefa93d41c0a2efe P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74077b61d6f54342bf9736b4a9f2eade, State: Initialized, Role: FOLLOWER
I20240501 07:59:12.505688 11385 tablet_bootstrap.cc:654] T 59e171892fc1414fbcfd6a8c090854f3 P 83f20e84af4c40d09a4804b6b26e697f: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:12.505759 11372 consensus_queue.cc:260] T 0864c290fafa488abefa93d41c0a2efe P 74077b61d6f54342bf9736b4a9f2eade [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } }
I20240501 07:59:12.506049 11372 ts_tablet_manager.cc:1398] T 0864c290fafa488abefa93d41c0a2efe P 74077b61d6f54342bf9736b4a9f2eade: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:12.506235 11372 tablet_bootstrap.cc:492] T 59e171892fc1414fbcfd6a8c090854f3 P 74077b61d6f54342bf9736b4a9f2eade: Bootstrap starting.
I20240501 07:59:12.506610 11371 tablet_bootstrap.cc:492] T 6daebcfc41754ec4aca939ca20588854 P a505e7cb133c4392bfda8a4a20874ae9: No bootstrap required, opened a new log
I20240501 07:59:12.506616 8904 tablet_service.cc:1467] Processing CreateTablet for tablet 59e171892fc1414fbcfd6a8c090854f3 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=731fe27ae8e34d92bd595d43c4bc8f3a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:12.506701 11371 ts_tablet_manager.cc:1367] T 6daebcfc41754ec4aca939ca20588854 P a505e7cb133c4392bfda8a4a20874ae9: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.001s
I20240501 07:59:12.506866 8904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59e171892fc1414fbcfd6a8c090854f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:12.507185 11371 raft_consensus.cc:348] T 6daebcfc41754ec4aca939ca20588854 P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:59:12.507311 11371 raft_consensus.cc:374] T 6daebcfc41754ec4aca939ca20588854 P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:12.507345 11371 raft_consensus.cc:729] T 6daebcfc41754ec4aca939ca20588854 P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a505e7cb133c4392bfda8a4a20874ae9, State: Initialized, Role: FOLLOWER
I20240501 07:59:12.507462 11371 consensus_queue.cc:260] T 6daebcfc41754ec4aca939ca20588854 P a505e7cb133c4392bfda8a4a20874ae9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:59:12.507576 8794 raft_consensus.cc:2932] T 239ee596e62d47169eeb6670dc62d59c P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { promote: true } } }
I20240501 07:59:12.507745 11371 ts_tablet_manager.cc:1398] T 6daebcfc41754ec4aca939ca20588854 P a505e7cb133c4392bfda8a4a20874ae9: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 07:59:12.508499 6169 catalog_manager.cc:5484] T 239ee596e62d47169eeb6670dc62d59c P 13d05ebdca8c4f489c597ce550dd773d reported cstate change: config changed from index -1 to 2, NON_VOTER 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132) added. New cstate: current_term: 1 leader_uuid: "13d05ebdca8c4f489c597ce550dd773d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 07:59:12.509582 11374 tablet_bootstrap.cc:492] T 59e171892fc1414fbcfd6a8c090854f3 P 13d05ebdca8c4f489c597ce550dd773d: Bootstrap starting.
I20240501 07:59:12.510085 11390 tablet_bootstrap.cc:492] T 6daebcfc41754ec4aca939ca20588854 P 1f9cb72be98941b5a8e504cbe34132b9: Bootstrap starting.
I20240501 07:59:12.510748 11374 tablet_bootstrap.cc:654] T 59e171892fc1414fbcfd6a8c090854f3 P 13d05ebdca8c4f489c597ce550dd773d: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:12.510941 11390 tablet_bootstrap.cc:654] T 6daebcfc41754ec4aca939ca20588854 P 1f9cb72be98941b5a8e504cbe34132b9: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:12.506616 8905 tablet_service.cc:1467] Processing CreateTablet for tablet 6daebcfc41754ec4aca939ca20588854 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=731fe27ae8e34d92bd595d43c4bc8f3a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:12.512197 8905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6daebcfc41754ec4aca939ca20588854. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:12.513096 9056 raft_consensus.cc:2932] T 239ee596e62d47169eeb6670dc62d59c P a505e7cb133c4392bfda8a4a20874ae9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { promote: true } } }
I20240501 07:59:12.513645 11374 tablet_bootstrap.cc:492] T 59e171892fc1414fbcfd6a8c090854f3 P 13d05ebdca8c4f489c597ce550dd773d: No bootstrap required, opened a new log
I20240501 07:59:12.513725 11374 ts_tablet_manager.cc:1367] T 59e171892fc1414fbcfd6a8c090854f3 P 13d05ebdca8c4f489c597ce550dd773d: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240501 07:59:12.514174 11374 raft_consensus.cc:348] T 59e171892fc1414fbcfd6a8c090854f3 P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } }
I20240501 07:59:12.514273 11374 raft_consensus.cc:374] T 59e171892fc1414fbcfd6a8c090854f3 P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:12.514325 11374 raft_consensus.cc:729] T 59e171892fc1414fbcfd6a8c090854f3 P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13d05ebdca8c4f489c597ce550dd773d, State: Initialized, Role: FOLLOWER
I20240501 07:59:12.514441 11374 consensus_queue.cc:260] T 59e171892fc1414fbcfd6a8c090854f3 P 13d05ebdca8c4f489c597ce550dd773d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } }
I20240501 07:59:12.514976 11374 ts_tablet_manager.cc:1398] T 59e171892fc1414fbcfd6a8c090854f3 P 13d05ebdca8c4f489c597ce550dd773d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:12.515154 11374 tablet_bootstrap.cc:492] T 6daebcfc41754ec4aca939ca20588854 P 13d05ebdca8c4f489c597ce550dd773d: Bootstrap starting.
I20240501 07:59:12.516206 11374 tablet_bootstrap.cc:654] T 6daebcfc41754ec4aca939ca20588854 P 13d05ebdca8c4f489c597ce550dd773d: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:12.517366 11374 tablet_bootstrap.cc:492] T 6daebcfc41754ec4aca939ca20588854 P 13d05ebdca8c4f489c597ce550dd773d: No bootstrap required, opened a new log
I20240501 07:59:12.517453 11374 ts_tablet_manager.cc:1367] T 6daebcfc41754ec4aca939ca20588854 P 13d05ebdca8c4f489c597ce550dd773d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.001s
I20240501 07:59:12.517499 11390 tablet_bootstrap.cc:492] T 6daebcfc41754ec4aca939ca20588854 P 1f9cb72be98941b5a8e504cbe34132b9: No bootstrap required, opened a new log
I20240501 07:59:12.517597 11390 ts_tablet_manager.cc:1367] T 6daebcfc41754ec4aca939ca20588854 P 1f9cb72be98941b5a8e504cbe34132b9: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20240501 07:59:12.517904 11374 raft_consensus.cc:348] T 6daebcfc41754ec4aca939ca20588854 P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:59:12.517999 11374 raft_consensus.cc:374] T 6daebcfc41754ec4aca939ca20588854 P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:12.518040 11374 raft_consensus.cc:729] T 6daebcfc41754ec4aca939ca20588854 P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13d05ebdca8c4f489c597ce550dd773d, State: Initialized, Role: FOLLOWER
I20240501 07:59:12.518168 11374 consensus_queue.cc:260] T 6daebcfc41754ec4aca939ca20588854 P 13d05ebdca8c4f489c597ce550dd773d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:59:12.518234 11390 raft_consensus.cc:348] T 6daebcfc41754ec4aca939ca20588854 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:59:12.518366 11390 raft_consensus.cc:374] T 6daebcfc41754ec4aca939ca20588854 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:12.518402 11390 raft_consensus.cc:729] T 6daebcfc41754ec4aca939ca20588854 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f9cb72be98941b5a8e504cbe34132b9, State: Initialized, Role: FOLLOWER
I20240501 07:59:12.518466 11374 ts_tablet_manager.cc:1398] T 6daebcfc41754ec4aca939ca20588854 P 13d05ebdca8c4f489c597ce550dd773d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:12.518527 11390 consensus_queue.cc:260] T 6daebcfc41754ec4aca939ca20588854 P 1f9cb72be98941b5a8e504cbe34132b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:59:12.518803 11390 ts_tablet_manager.cc:1398] T 6daebcfc41754ec4aca939ca20588854 P 1f9cb72be98941b5a8e504cbe34132b9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:12.519445 11372 tablet_bootstrap.cc:654] T 59e171892fc1414fbcfd6a8c090854f3 P 74077b61d6f54342bf9736b4a9f2eade: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:12.520771 11372 tablet_bootstrap.cc:492] T 59e171892fc1414fbcfd6a8c090854f3 P 74077b61d6f54342bf9736b4a9f2eade: No bootstrap required, opened a new log
I20240501 07:59:12.520874 11372 ts_tablet_manager.cc:1367] T 59e171892fc1414fbcfd6a8c090854f3 P 74077b61d6f54342bf9736b4a9f2eade: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.000s
I20240501 07:59:12.521394 11372 raft_consensus.cc:348] T 59e171892fc1414fbcfd6a8c090854f3 P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } }
I20240501 07:59:12.521500 11372 raft_consensus.cc:374] T 59e171892fc1414fbcfd6a8c090854f3 P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:12.521539 11372 raft_consensus.cc:729] T 59e171892fc1414fbcfd6a8c090854f3 P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74077b61d6f54342bf9736b4a9f2eade, State: Initialized, Role: FOLLOWER
I20240501 07:59:12.521661 11372 consensus_queue.cc:260] T 59e171892fc1414fbcfd6a8c090854f3 P 74077b61d6f54342bf9736b4a9f2eade [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } }
I20240501 07:59:12.521956 11372 ts_tablet_manager.cc:1398] T 59e171892fc1414fbcfd6a8c090854f3 P 74077b61d6f54342bf9736b4a9f2eade: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:59:12.522483 11385 tablet_bootstrap.cc:492] T 59e171892fc1414fbcfd6a8c090854f3 P 83f20e84af4c40d09a4804b6b26e697f: No bootstrap required, opened a new log
I20240501 07:59:12.522581 11385 ts_tablet_manager.cc:1367] T 59e171892fc1414fbcfd6a8c090854f3 P 83f20e84af4c40d09a4804b6b26e697f: Time spent bootstrapping tablet: real 0.024s user 0.002s sys 0.000s
I20240501 07:59:12.523015 11385 raft_consensus.cc:348] T 59e171892fc1414fbcfd6a8c090854f3 P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } }
I20240501 07:59:12.523118 11385 raft_consensus.cc:374] T 59e171892fc1414fbcfd6a8c090854f3 P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:12.523149 11385 raft_consensus.cc:729] T 59e171892fc1414fbcfd6a8c090854f3 P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83f20e84af4c40d09a4804b6b26e697f, State: Initialized, Role: FOLLOWER
I20240501 07:59:12.523265 11385 consensus_queue.cc:260] T 59e171892fc1414fbcfd6a8c090854f3 P 83f20e84af4c40d09a4804b6b26e697f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } }
I20240501 07:59:12.523515 11385 ts_tablet_manager.cc:1398] T 59e171892fc1414fbcfd6a8c090854f3 P 83f20e84af4c40d09a4804b6b26e697f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:12.523663 11385 tablet_bootstrap.cc:492] T 0864c290fafa488abefa93d41c0a2efe P 83f20e84af4c40d09a4804b6b26e697f: Bootstrap starting.
I20240501 07:59:12.524870 11385 tablet_bootstrap.cc:654] T 0864c290fafa488abefa93d41c0a2efe P 83f20e84af4c40d09a4804b6b26e697f: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:12.526985 11385 tablet_bootstrap.cc:492] T 0864c290fafa488abefa93d41c0a2efe P 83f20e84af4c40d09a4804b6b26e697f: No bootstrap required, opened a new log
I20240501 07:59:12.527128 11385 ts_tablet_manager.cc:1367] T 0864c290fafa488abefa93d41c0a2efe P 83f20e84af4c40d09a4804b6b26e697f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 07:59:12.527570 11385 raft_consensus.cc:348] T 0864c290fafa488abefa93d41c0a2efe P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } }
I20240501 07:59:12.527686 11385 raft_consensus.cc:374] T 0864c290fafa488abefa93d41c0a2efe P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:12.527722 11385 raft_consensus.cc:729] T 0864c290fafa488abefa93d41c0a2efe P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83f20e84af4c40d09a4804b6b26e697f, State: Initialized, Role: FOLLOWER
I20240501 07:59:12.528095 11385 consensus_queue.cc:260] T 0864c290fafa488abefa93d41c0a2efe P 83f20e84af4c40d09a4804b6b26e697f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } }
I20240501 07:59:12.528400 11385 ts_tablet_manager.cc:1398] T 0864c290fafa488abefa93d41c0a2efe P 83f20e84af4c40d09a4804b6b26e697f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:12.530566 6707 consensus_queue.cc:237] T 8663a511521644009fc360ad92c093a1 P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: true } }
I20240501 07:59:12.531985 6444 raft_consensus.cc:1261] T 8663a511521644009fc360ad92c093a1 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Refusing update from remote peer 1f9cb72be98941b5a8e504cbe34132b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:59:12.532065 6577 raft_consensus.cc:1261] T 8663a511521644009fc360ad92c093a1 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Refusing update from remote peer 1f9cb72be98941b5a8e504cbe34132b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 07:59:12.532096 6643 consensus_peers.cc:485] T 8663a511521644009fc360ad92c093a1 P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134:40141): Couldn't send request to peer 13d05ebdca8c4f489c597ce550dd773d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8663a511521644009fc360ad92c093a1. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:12.532481 10324 consensus_queue.cc:1035] T 8663a511521644009fc360ad92c093a1 P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:12.532634 11402 consensus_queue.cc:1035] T 8663a511521644009fc360ad92c093a1 P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:12.534807 11402 raft_consensus.cc:2932] T 8663a511521644009fc360ad92c093a1 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: true } } }
I20240501 07:59:12.535218 6577 raft_consensus.cc:2932] T 8663a511521644009fc360ad92c093a1 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: true } } }
I20240501 07:59:12.536828 6444 raft_consensus.cc:2932] T 8663a511521644009fc360ad92c093a1 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: true } } }
I20240501 07:59:12.538116 6171 catalog_manager.cc:5484] T 8663a511521644009fc360ad92c093a1 P 3b164cbbc00f4e8eb3254ddd349a3ff9 reported cstate change: config changed from index -1 to 4, NON_VOTER 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "1f9cb72be98941b5a8e504cbe34132b9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: true } } }
I20240501 07:59:12.539003 10478 raft_consensus.cc:482] T 0864c290fafa488abefa93d41c0a2efe P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:12.539119 10478 raft_consensus.cc:504] T 0864c290fafa488abefa93d41c0a2efe P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } }
I20240501 07:59:12.539505 10478 leader_election.cc:288] T 0864c290fafa488abefa93d41c0a2efe P 13d05ebdca8c4f489c597ce550dd773d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149), 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133:39859)
I20240501 07:59:12.539959 9189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0864c290fafa488abefa93d41c0a2efe" candidate_uuid: "13d05ebdca8c4f489c597ce550dd773d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83f20e84af4c40d09a4804b6b26e697f" is_pre_election: true
I20240501 07:59:12.540023 8794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0864c290fafa488abefa93d41c0a2efe" candidate_uuid: "13d05ebdca8c4f489c597ce550dd773d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74077b61d6f54342bf9736b4a9f2eade" is_pre_election: true
I20240501 07:59:12.540091 9189 raft_consensus.cc:2454] T 0864c290fafa488abefa93d41c0a2efe P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 13d05ebdca8c4f489c597ce550dd773d in term 0.
I20240501 07:59:12.540151 8794 raft_consensus.cc:2454] T 0864c290fafa488abefa93d41c0a2efe P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 13d05ebdca8c4f489c597ce550dd773d in term 0.
I20240501 07:59:12.540566 8859 leader_election.cc:302] T 0864c290fafa488abefa93d41c0a2efe P 13d05ebdca8c4f489c597ce550dd773d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13d05ebdca8c4f489c597ce550dd773d, 83f20e84af4c40d09a4804b6b26e697f; no voters:
I20240501 07:59:12.541256 10478 raft_consensus.cc:2781] T 0864c290fafa488abefa93d41c0a2efe P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:12.541316 10478 raft_consensus.cc:482] T 0864c290fafa488abefa93d41c0a2efe P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:12.541357 10478 raft_consensus.cc:3037] T 0864c290fafa488abefa93d41c0a2efe P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:12.542322 10478 raft_consensus.cc:504] T 0864c290fafa488abefa93d41c0a2efe P 13d05ebdca8c4f489c597ce550dd773d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } }
I20240501 07:59:12.542651 10478 leader_election.cc:288] T 0864c290fafa488abefa93d41c0a2efe P 13d05ebdca8c4f489c597ce550dd773d [CANDIDATE]: Term 1 election: Requested vote from peers 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149), 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133:39859)
I20240501 07:59:12.542979 9189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0864c290fafa488abefa93d41c0a2efe" candidate_uuid: "13d05ebdca8c4f489c597ce550dd773d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83f20e84af4c40d09a4804b6b26e697f"
I20240501 07:59:12.543105 9189 raft_consensus.cc:3037] T 0864c290fafa488abefa93d41c0a2efe P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:12.543102 8794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0864c290fafa488abefa93d41c0a2efe" candidate_uuid: "13d05ebdca8c4f489c597ce550dd773d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74077b61d6f54342bf9736b4a9f2eade"
I20240501 07:59:12.543212 8794 raft_consensus.cc:3037] T 0864c290fafa488abefa93d41c0a2efe P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:12.544077 9189 raft_consensus.cc:2454] T 0864c290fafa488abefa93d41c0a2efe P 83f20e84af4c40d09a4804b6b26e697f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13d05ebdca8c4f489c597ce550dd773d in term 1.
I20240501 07:59:12.544080 8794 raft_consensus.cc:2454] T 0864c290fafa488abefa93d41c0a2efe P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13d05ebdca8c4f489c597ce550dd773d in term 1.
I20240501 07:59:12.544343 8858 leader_election.cc:302] T 0864c290fafa488abefa93d41c0a2efe P 13d05ebdca8c4f489c597ce550dd773d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13d05ebdca8c4f489c597ce550dd773d, 74077b61d6f54342bf9736b4a9f2eade; no voters:
I20240501 07:59:12.544540 10478 raft_consensus.cc:2781] T 0864c290fafa488abefa93d41c0a2efe P 13d05ebdca8c4f489c597ce550dd773d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:12.544617 10478 raft_consensus.cc:686] T 0864c290fafa488abefa93d41c0a2efe P 13d05ebdca8c4f489c597ce550dd773d [term 1 LEADER]: Becoming Leader. State: Replica: 13d05ebdca8c4f489c597ce550dd773d, State: Running, Role: LEADER
I20240501 07:59:12.544756 10478 consensus_queue.cc:237] T 0864c290fafa488abefa93d41c0a2efe P 13d05ebdca8c4f489c597ce550dd773d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } }
I20240501 07:59:12.546554 6169 catalog_manager.cc:5484] T 0864c290fafa488abefa93d41c0a2efe P 13d05ebdca8c4f489c597ce550dd773d reported cstate change: term changed from 0 to 1, leader changed from <none> to 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134). New cstate: current_term: 1 leader_uuid: "13d05ebdca8c4f489c597ce550dd773d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } health_report { overall_health: UNKNOWN } } }
I20240501 07:59:12.552382 6444 consensus_queue.cc:237] T a756205204d146ce82a2799e30582f57 P 8315d22c56f7439a9e1fc82f153f018c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: true } }
I20240501 07:59:12.553951 6577 raft_consensus.cc:1261] T a756205204d146ce82a2799e30582f57 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Refusing update from remote peer 8315d22c56f7439a9e1fc82f153f018c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 07:59:12.553956 6380 consensus_peers.cc:485] T a756205204d146ce82a2799e30582f57 P 8315d22c56f7439a9e1fc82f153f018c -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a756205204d146ce82a2799e30582f57. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:12.554082 6707 raft_consensus.cc:1261] T a756205204d146ce82a2799e30582f57 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Refusing update from remote peer 8315d22c56f7439a9e1fc82f153f018c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:59:12.554472 9116 consensus_queue.cc:1035] T a756205204d146ce82a2799e30582f57 P 8315d22c56f7439a9e1fc82f153f018c [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:12.554677 9489 consensus_queue.cc:1035] T a756205204d146ce82a2799e30582f57 P 8315d22c56f7439a9e1fc82f153f018c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:12.556562 9116 raft_consensus.cc:2932] T a756205204d146ce82a2799e30582f57 P 8315d22c56f7439a9e1fc82f153f018c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: true } } }
I20240501 07:59:12.557134 6577 raft_consensus.cc:2932] T a756205204d146ce82a2799e30582f57 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: true } } }
I20240501 07:59:12.557354 6707 raft_consensus.cc:2932] T a756205204d146ce82a2799e30582f57 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: true } } }
I20240501 07:59:12.559087 11402 raft_consensus.cc:482] T 6daebcfc41754ec4aca939ca20588854 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:12.559167 11402 raft_consensus.cc:504] T 6daebcfc41754ec4aca939ca20588854 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:59:12.559507 11402 leader_election.cc:288] T 6daebcfc41754ec4aca939ca20588854 P 1f9cb72be98941b5a8e504cbe34132b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134:40141), a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607)
I20240501 07:59:12.559854 9056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6daebcfc41754ec4aca939ca20588854" candidate_uuid: "1f9cb72be98941b5a8e504cbe34132b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a505e7cb133c4392bfda8a4a20874ae9" is_pre_election: true
I20240501 07:59:12.559996 9056 raft_consensus.cc:2454] T 6daebcfc41754ec4aca939ca20588854 P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f9cb72be98941b5a8e504cbe34132b9 in term 0.
I20240501 07:59:12.559964 8924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6daebcfc41754ec4aca939ca20588854" candidate_uuid: "1f9cb72be98941b5a8e504cbe34132b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13d05ebdca8c4f489c597ce550dd773d" is_pre_election: true
I20240501 07:59:12.560092 8924 raft_consensus.cc:2454] T 6daebcfc41754ec4aca939ca20588854 P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f9cb72be98941b5a8e504cbe34132b9 in term 0.
I20240501 07:59:12.560560 6643 leader_election.cc:302] T 6daebcfc41754ec4aca939ca20588854 P 1f9cb72be98941b5a8e504cbe34132b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13d05ebdca8c4f489c597ce550dd773d, 1f9cb72be98941b5a8e504cbe34132b9; no voters:
I20240501 07:59:12.560524 6166 catalog_manager.cc:5484] T a756205204d146ce82a2799e30582f57 P 3b164cbbc00f4e8eb3254ddd349a3ff9 reported cstate change: config changed from index -1 to 4, NON_VOTER a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "8315d22c56f7439a9e1fc82f153f018c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: true } } }
I20240501 07:59:12.562390 11402 raft_consensus.cc:2781] T 6daebcfc41754ec4aca939ca20588854 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:12.562455 11402 raft_consensus.cc:482] T 6daebcfc41754ec4aca939ca20588854 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:12.562494 11402 raft_consensus.cc:3037] T 6daebcfc41754ec4aca939ca20588854 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:12.563459 11402 raft_consensus.cc:504] T 6daebcfc41754ec4aca939ca20588854 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:59:12.563808 11402 leader_election.cc:288] T 6daebcfc41754ec4aca939ca20588854 P 1f9cb72be98941b5a8e504cbe34132b9 [CANDIDATE]: Term 1 election: Requested vote from peers 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134:40141), a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607)
I20240501 07:59:12.564172 9055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6daebcfc41754ec4aca939ca20588854" candidate_uuid: "1f9cb72be98941b5a8e504cbe34132b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a505e7cb133c4392bfda8a4a20874ae9"
I20240501 07:59:12.564318 9055 raft_consensus.cc:3037] T 6daebcfc41754ec4aca939ca20588854 P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:12.565428 9055 raft_consensus.cc:2454] T 6daebcfc41754ec4aca939ca20588854 P a505e7cb133c4392bfda8a4a20874ae9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f9cb72be98941b5a8e504cbe34132b9 in term 1.
I20240501 07:59:12.565688 6642 leader_election.cc:302] T 6daebcfc41754ec4aca939ca20588854 P 1f9cb72be98941b5a8e504cbe34132b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f9cb72be98941b5a8e504cbe34132b9, a505e7cb133c4392bfda8a4a20874ae9; no voters:
I20240501 07:59:12.565732 11317 ts_tablet_manager.cc:902] T d3f2250a62c6429e886753aaffcbe883 P a505e7cb133c4392bfda8a4a20874ae9: Initiating tablet copy from peer 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131:34471)
I20240501 07:59:12.565862 11402 raft_consensus.cc:2781] T 6daebcfc41754ec4aca939ca20588854 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:12.565968 11402 raft_consensus.cc:686] T 6daebcfc41754ec4aca939ca20588854 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 LEADER]: Becoming Leader. State: Replica: 1f9cb72be98941b5a8e504cbe34132b9, State: Running, Role: LEADER
I20240501 07:59:12.566152 11402 consensus_queue.cc:237] T 6daebcfc41754ec4aca939ca20588854 P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:59:12.566311 11317 tablet_copy_client.cc:308] T d3f2250a62c6429e886753aaffcbe883 P a505e7cb133c4392bfda8a4a20874ae9: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:34471
I20240501 07:59:12.566709 8924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6daebcfc41754ec4aca939ca20588854" candidate_uuid: "1f9cb72be98941b5a8e504cbe34132b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13d05ebdca8c4f489c597ce550dd773d"
I20240501 07:59:12.566831 8924 raft_consensus.cc:3037] T 6daebcfc41754ec4aca939ca20588854 P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:12.566854 6597 tablet_copy_service.cc:140] P 3b164cbbc00f4e8eb3254ddd349a3ff9: Received BeginTabletCopySession request for tablet d3f2250a62c6429e886753aaffcbe883 from peer a505e7cb133c4392bfda8a4a20874ae9 ({username='slave'} at 127.0.99.135:40821)
I20240501 07:59:12.566972 6597 tablet_copy_service.cc:161] P 3b164cbbc00f4e8eb3254ddd349a3ff9: Beginning new tablet copy session on tablet d3f2250a62c6429e886753aaffcbe883 from peer a505e7cb133c4392bfda8a4a20874ae9 at {username='slave'} at 127.0.99.135:40821: session id = a505e7cb133c4392bfda8a4a20874ae9-d3f2250a62c6429e886753aaffcbe883
I20240501 07:59:12.567785 8924 raft_consensus.cc:2454] T 6daebcfc41754ec4aca939ca20588854 P 13d05ebdca8c4f489c597ce550dd773d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f9cb72be98941b5a8e504cbe34132b9 in term 1.
I20240501 07:59:12.568014 6597 tablet_copy_source_session.cc:205] T d3f2250a62c6429e886753aaffcbe883 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:12.568924 11317 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3f2250a62c6429e886753aaffcbe883. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:12.569332 6171 catalog_manager.cc:5484] T 6daebcfc41754ec4aca939ca20588854 P 1f9cb72be98941b5a8e504cbe34132b9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132). New cstate: current_term: 1 leader_uuid: "1f9cb72be98941b5a8e504cbe34132b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } health_report { overall_health: HEALTHY } } }
I20240501 07:59:12.571010 11317 tablet_copy_client.cc:783] T d3f2250a62c6429e886753aaffcbe883 P a505e7cb133c4392bfda8a4a20874ae9: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:12.571164 11317 tablet_copy_client.cc:647] T d3f2250a62c6429e886753aaffcbe883 P a505e7cb133c4392bfda8a4a20874ae9: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:12.573019 11317 tablet_copy_client.cc:523] T d3f2250a62c6429e886753aaffcbe883 P a505e7cb133c4392bfda8a4a20874ae9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:12.573634 6707 consensus_queue.cc:237] T ff9dc384dbac47dc9795617c64688943 P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { replace: true } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: true } }
I20240501 07:59:12.574119 11317 tablet_bootstrap.cc:492] T d3f2250a62c6429e886753aaffcbe883 P a505e7cb133c4392bfda8a4a20874ae9: Bootstrap starting.
I20240501 07:59:12.575212 8924 raft_consensus.cc:1261] T ff9dc384dbac47dc9795617c64688943 P 13d05ebdca8c4f489c597ce550dd773d [term 2 FOLLOWER]: Refusing update from remote peer 1f9cb72be98941b5a8e504cbe34132b9: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240501 07:59:12.575510 6313 raft_consensus.cc:1261] T ff9dc384dbac47dc9795617c64688943 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 2 FOLLOWER]: Refusing update from remote peer 1f9cb72be98941b5a8e504cbe34132b9: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240501 07:59:12.575640 11402 consensus_queue.cc:1035] T ff9dc384dbac47dc9795617c64688943 P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240501 07:59:12.575834 6642 consensus_peers.cc:485] T ff9dc384dbac47dc9795617c64688943 P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ff9dc384dbac47dc9795617c64688943. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:12.576239 11402 consensus_queue.cc:1035] T ff9dc384dbac47dc9795617c64688943 P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 07:59:12.578181 11171 raft_consensus.cc:2932] T ff9dc384dbac47dc9795617c64688943 P 1f9cb72be98941b5a8e504cbe34132b9 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { replace: true } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: true } } }
I20240501 07:59:12.578883 8924 raft_consensus.cc:2932] T ff9dc384dbac47dc9795617c64688943 P 13d05ebdca8c4f489c597ce550dd773d [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { replace: true } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: true } } }
I20240501 07:59:12.580054 6313 raft_consensus.cc:2932] T ff9dc384dbac47dc9795617c64688943 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { replace: true } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: true } } }
I20240501 07:59:12.581996 6169 catalog_manager.cc:5484] T ff9dc384dbac47dc9795617c64688943 P 13d05ebdca8c4f489c597ce550dd773d reported cstate change: config changed from index 7 to 8, NON_VOTER 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136) added. New cstate: current_term: 2 leader_uuid: "1f9cb72be98941b5a8e504cbe34132b9" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { replace: true } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: true } } }
I20240501 07:59:12.583323 10945 raft_consensus.cc:482] T 32ecf214d6e14fa3be1b8eafd8c23fac P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:12.583411 10945 raft_consensus.cc:504] T 32ecf214d6e14fa3be1b8eafd8c23fac P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } }
I20240501 07:59:12.583758 10945 leader_election.cc:288] T 32ecf214d6e14fa3be1b8eafd8c23fac P a505e7cb133c4392bfda8a4a20874ae9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134:40141), 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133:39859)
I20240501 07:59:12.584934 8793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32ecf214d6e14fa3be1b8eafd8c23fac" candidate_uuid: "a505e7cb133c4392bfda8a4a20874ae9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74077b61d6f54342bf9736b4a9f2eade" is_pre_election: true
I20240501 07:59:12.585134 8793 raft_consensus.cc:2454] T 32ecf214d6e14fa3be1b8eafd8c23fac P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a505e7cb133c4392bfda8a4a20874ae9 in term 0.
I20240501 07:59:12.585947 8989 leader_election.cc:302] T 32ecf214d6e14fa3be1b8eafd8c23fac P a505e7cb133c4392bfda8a4a20874ae9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74077b61d6f54342bf9736b4a9f2eade, a505e7cb133c4392bfda8a4a20874ae9; no voters:
I20240501 07:59:12.586105 10945 raft_consensus.cc:2781] T 32ecf214d6e14fa3be1b8eafd8c23fac P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:12.586159 10945 raft_consensus.cc:482] T 32ecf214d6e14fa3be1b8eafd8c23fac P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:12.586201 10945 raft_consensus.cc:3037] T 32ecf214d6e14fa3be1b8eafd8c23fac P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:12.586287 8924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32ecf214d6e14fa3be1b8eafd8c23fac" candidate_uuid: "a505e7cb133c4392bfda8a4a20874ae9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13d05ebdca8c4f489c597ce550dd773d" is_pre_election: true
I20240501 07:59:12.586426 8924 raft_consensus.cc:2454] T 32ecf214d6e14fa3be1b8eafd8c23fac P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a505e7cb133c4392bfda8a4a20874ae9 in term 0.
I20240501 07:59:12.587244 10945 raft_consensus.cc:504] T 32ecf214d6e14fa3be1b8eafd8c23fac P a505e7cb133c4392bfda8a4a20874ae9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } }
I20240501 07:59:12.587611 10945 leader_election.cc:288] T 32ecf214d6e14fa3be1b8eafd8c23fac P a505e7cb133c4392bfda8a4a20874ae9 [CANDIDATE]: Term 1 election: Requested vote from peers 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134:40141), 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133:39859)
I20240501 07:59:12.588821 8924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32ecf214d6e14fa3be1b8eafd8c23fac" candidate_uuid: "a505e7cb133c4392bfda8a4a20874ae9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13d05ebdca8c4f489c597ce550dd773d"
I20240501 07:59:12.588932 8924 raft_consensus.cc:3037] T 32ecf214d6e14fa3be1b8eafd8c23fac P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:12.589773 8924 raft_consensus.cc:2454] T 32ecf214d6e14fa3be1b8eafd8c23fac P 13d05ebdca8c4f489c597ce550dd773d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a505e7cb133c4392bfda8a4a20874ae9 in term 1.
I20240501 07:59:12.590056 8991 leader_election.cc:302] T 32ecf214d6e14fa3be1b8eafd8c23fac P a505e7cb133c4392bfda8a4a20874ae9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13d05ebdca8c4f489c597ce550dd773d, a505e7cb133c4392bfda8a4a20874ae9; no voters:
I20240501 07:59:12.590214 10945 raft_consensus.cc:2781] T 32ecf214d6e14fa3be1b8eafd8c23fac P a505e7cb133c4392bfda8a4a20874ae9 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:12.590283 10945 raft_consensus.cc:686] T 32ecf214d6e14fa3be1b8eafd8c23fac P a505e7cb133c4392bfda8a4a20874ae9 [term 1 LEADER]: Becoming Leader. State: Replica: a505e7cb133c4392bfda8a4a20874ae9, State: Running, Role: LEADER
I20240501 07:59:12.590407 10945 consensus_queue.cc:237] T 32ecf214d6e14fa3be1b8eafd8c23fac P a505e7cb133c4392bfda8a4a20874ae9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } }
I20240501 07:59:12.591163 8794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32ecf214d6e14fa3be1b8eafd8c23fac" candidate_uuid: "a505e7cb133c4392bfda8a4a20874ae9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74077b61d6f54342bf9736b4a9f2eade"
I20240501 07:59:12.591277 8794 raft_consensus.cc:3037] T 32ecf214d6e14fa3be1b8eafd8c23fac P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:12.592271 8794 raft_consensus.cc:2454] T 32ecf214d6e14fa3be1b8eafd8c23fac P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a505e7cb133c4392bfda8a4a20874ae9 in term 1.
I20240501 07:59:12.593045 6166 catalog_manager.cc:5484] T 32ecf214d6e14fa3be1b8eafd8c23fac P a505e7cb133c4392bfda8a4a20874ae9 reported cstate change: term changed from 0 to 1, leader changed from <none> to a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135). New cstate: current_term: 1 leader_uuid: "a505e7cb133c4392bfda8a4a20874ae9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } health_report { overall_health: UNKNOWN } } }
I20240501 07:59:12.593313 10478 raft_consensus.cc:482] T 59e171892fc1414fbcfd6a8c090854f3 P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:12.593385 10478 raft_consensus.cc:504] T 59e171892fc1414fbcfd6a8c090854f3 P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } }
I20240501 07:59:12.593679 10478 leader_election.cc:288] T 59e171892fc1414fbcfd6a8c090854f3 P 13d05ebdca8c4f489c597ce550dd773d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149), 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133:39859)
I20240501 07:59:12.594029 9189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59e171892fc1414fbcfd6a8c090854f3" candidate_uuid: "13d05ebdca8c4f489c597ce550dd773d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83f20e84af4c40d09a4804b6b26e697f" is_pre_election: true
I20240501 07:59:12.594163 9189 raft_consensus.cc:2454] T 59e171892fc1414fbcfd6a8c090854f3 P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 13d05ebdca8c4f489c597ce550dd773d in term 0.
I20240501 07:59:12.594161 8794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59e171892fc1414fbcfd6a8c090854f3" candidate_uuid: "13d05ebdca8c4f489c597ce550dd773d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74077b61d6f54342bf9736b4a9f2eade" is_pre_election: true
I20240501 07:59:12.594274 8794 raft_consensus.cc:2454] T 59e171892fc1414fbcfd6a8c090854f3 P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 13d05ebdca8c4f489c597ce550dd773d in term 0.
I20240501 07:59:12.594561 8859 leader_election.cc:302] T 59e171892fc1414fbcfd6a8c090854f3 P 13d05ebdca8c4f489c597ce550dd773d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13d05ebdca8c4f489c597ce550dd773d, 83f20e84af4c40d09a4804b6b26e697f; no voters:
I20240501 07:59:12.595029 10478 raft_consensus.cc:2781] T 59e171892fc1414fbcfd6a8c090854f3 P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:12.595091 10478 raft_consensus.cc:482] T 59e171892fc1414fbcfd6a8c090854f3 P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:12.595136 10478 raft_consensus.cc:3037] T 59e171892fc1414fbcfd6a8c090854f3 P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:12.596145 10478 raft_consensus.cc:504] T 59e171892fc1414fbcfd6a8c090854f3 P 13d05ebdca8c4f489c597ce550dd773d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } }
I20240501 07:59:12.596457 10478 leader_election.cc:288] T 59e171892fc1414fbcfd6a8c090854f3 P 13d05ebdca8c4f489c597ce550dd773d [CANDIDATE]: Term 1 election: Requested vote from peers 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149), 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133:39859)
I20240501 07:59:12.596899 9189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59e171892fc1414fbcfd6a8c090854f3" candidate_uuid: "13d05ebdca8c4f489c597ce550dd773d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83f20e84af4c40d09a4804b6b26e697f"
I20240501 07:59:12.597040 9189 raft_consensus.cc:3037] T 59e171892fc1414fbcfd6a8c090854f3 P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:12.598016 9189 raft_consensus.cc:2454] T 59e171892fc1414fbcfd6a8c090854f3 P 83f20e84af4c40d09a4804b6b26e697f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13d05ebdca8c4f489c597ce550dd773d in term 1.
I20240501 07:59:12.598281 8859 leader_election.cc:302] T 59e171892fc1414fbcfd6a8c090854f3 P 13d05ebdca8c4f489c597ce550dd773d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13d05ebdca8c4f489c597ce550dd773d, 83f20e84af4c40d09a4804b6b26e697f; no voters:
I20240501 07:59:12.598423 10478 raft_consensus.cc:2781] T 59e171892fc1414fbcfd6a8c090854f3 P 13d05ebdca8c4f489c597ce550dd773d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:12.598491 10478 raft_consensus.cc:686] T 59e171892fc1414fbcfd6a8c090854f3 P 13d05ebdca8c4f489c597ce550dd773d [term 1 LEADER]: Becoming Leader. State: Replica: 13d05ebdca8c4f489c597ce550dd773d, State: Running, Role: LEADER
I20240501 07:59:12.598606 10478 consensus_queue.cc:237] T 59e171892fc1414fbcfd6a8c090854f3 P 13d05ebdca8c4f489c597ce550dd773d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } }
I20240501 07:59:12.599118 8794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59e171892fc1414fbcfd6a8c090854f3" candidate_uuid: "13d05ebdca8c4f489c597ce550dd773d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74077b61d6f54342bf9736b4a9f2eade"
I20240501 07:59:12.599251 8794 raft_consensus.cc:3037] T 59e171892fc1414fbcfd6a8c090854f3 P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:12.600114 8794 raft_consensus.cc:2454] T 59e171892fc1414fbcfd6a8c090854f3 P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13d05ebdca8c4f489c597ce550dd773d in term 1.
I20240501 07:59:12.601492 6577 consensus_queue.cc:237] T d6c28cd91f624673b8e428f471a1dcce P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { replace: true } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: true } }
I20240501 07:59:12.601850 6166 catalog_manager.cc:5484] T 59e171892fc1414fbcfd6a8c090854f3 P 13d05ebdca8c4f489c597ce550dd773d reported cstate change: term changed from 0 to 1, leader changed from <none> to 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134). New cstate: current_term: 1 leader_uuid: "13d05ebdca8c4f489c597ce550dd773d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } health_report { overall_health: HEALTHY } } }
I20240501 07:59:12.602892 6313 raft_consensus.cc:1261] T d6c28cd91f624673b8e428f471a1dcce P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Refusing update from remote peer 3b164cbbc00f4e8eb3254ddd349a3ff9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:59:12.603286 6444 raft_consensus.cc:1261] T d6c28cd91f624673b8e428f471a1dcce P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Refusing update from remote peer 3b164cbbc00f4e8eb3254ddd349a3ff9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:59:12.603328 11107 consensus_queue.cc:1035] T d6c28cd91f624673b8e428f471a1dcce P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 07:59:12.603706 6511 consensus_peers.cc:485] T d6c28cd91f624673b8e428f471a1dcce P 3b164cbbc00f4e8eb3254ddd349a3ff9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d6c28cd91f624673b8e428f471a1dcce. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:12.603713 11107 consensus_queue.cc:1035] T d6c28cd91f624673b8e428f471a1dcce P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:12.606005 11119 raft_consensus.cc:2932] T d6c28cd91f624673b8e428f471a1dcce P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { replace: true } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: true } } }
I20240501 07:59:12.606539 6313 raft_consensus.cc:2932] T d6c28cd91f624673b8e428f471a1dcce P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { replace: true } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: true } } }
I20240501 07:59:12.608417 6444 raft_consensus.cc:2932] T d6c28cd91f624673b8e428f471a1dcce P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { replace: true } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: true } } }
I20240501 07:59:12.608836 6166 catalog_manager.cc:5484] T d6c28cd91f624673b8e428f471a1dcce P 8b8b3ad3aa8f4053b2167dd550f2086f reported cstate change: config changed from index -1 to 4, NON_VOTER a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { replace: true } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: true } } }
I20240501 07:59:12.610553 11317 tablet_bootstrap.cc:492] T d3f2250a62c6429e886753aaffcbe883 P a505e7cb133c4392bfda8a4a20874ae9: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:12.610834 11317 tablet_bootstrap.cc:492] T d3f2250a62c6429e886753aaffcbe883 P a505e7cb133c4392bfda8a4a20874ae9: Bootstrap complete.
I20240501 07:59:12.611016 11317 ts_tablet_manager.cc:1367] T d3f2250a62c6429e886753aaffcbe883 P a505e7cb133c4392bfda8a4a20874ae9: Time spent bootstrapping tablet: real 0.037s user 0.027s sys 0.003s
I20240501 07:59:12.611521 11317 raft_consensus.cc:348] T d3f2250a62c6429e886753aaffcbe883 P a505e7cb133c4392bfda8a4a20874ae9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { replace: true } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: true } }
I20240501 07:59:12.611653 11317 raft_consensus.cc:729] T d3f2250a62c6429e886753aaffcbe883 P a505e7cb133c4392bfda8a4a20874ae9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a505e7cb133c4392bfda8a4a20874ae9, State: Initialized, Role: LEARNER
I20240501 07:59:12.611763 11317 consensus_queue.cc:260] T d3f2250a62c6429e886753aaffcbe883 P a505e7cb133c4392bfda8a4a20874ae9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { replace: true } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: true } }
I20240501 07:59:12.612210 11317 ts_tablet_manager.cc:1398] T d3f2250a62c6429e886753aaffcbe883 P a505e7cb133c4392bfda8a4a20874ae9: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20240501 07:59:12.612751 6597 tablet_copy_service.cc:342] P 3b164cbbc00f4e8eb3254ddd349a3ff9: Request end of tablet copy session a505e7cb133c4392bfda8a4a20874ae9-d3f2250a62c6429e886753aaffcbe883 received from {username='slave'} at 127.0.99.135:40821
I20240501 07:59:12.612836 6597 tablet_copy_service.cc:433] P 3b164cbbc00f4e8eb3254ddd349a3ff9: ending tablet copy session a505e7cb133c4392bfda8a4a20874ae9-d3f2250a62c6429e886753aaffcbe883 on tablet d3f2250a62c6429e886753aaffcbe883 with peer a505e7cb133c4392bfda8a4a20874ae9
I20240501 07:59:12.638443 6577 raft_consensus.cc:1203] T 2976d6353e8947b7b6c37f72efbeeac4 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 07:59:12.640508 6444 raft_consensus.cc:1203] T 72aa4dc7664b48b7a73d34feba7d44d3 P 8315d22c56f7439a9e1fc82f153f018c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 07:59:12.726943 6171 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55950:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 07:59:12.731839 8794 raft_consensus.cc:1261] T 32ecf214d6e14fa3be1b8eafd8c23fac P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Refusing update from remote peer a505e7cb133c4392bfda8a4a20874ae9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:12.731933 8924 raft_consensus.cc:1261] T 32ecf214d6e14fa3be1b8eafd8c23fac P 13d05ebdca8c4f489c597ce550dd773d [term 1 FOLLOWER]: Refusing update from remote peer a505e7cb133c4392bfda8a4a20874ae9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:12.732489 10945 consensus_queue.cc:1035] T 32ecf214d6e14fa3be1b8eafd8c23fac P a505e7cb133c4392bfda8a4a20874ae9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:12.732653 11453 consensus_queue.cc:1035] T 32ecf214d6e14fa3be1b8eafd8c23fac P a505e7cb133c4392bfda8a4a20874ae9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:12.736073 6171 catalog_manager.cc:5576] TS 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134:40141) does not have the latest schema for tablet 32ecf214d6e14fa3be1b8eafd8c23fac (table rebalancer_dynamic_table [id=95208beb4f5944b09e76c37f268c35ba]). Expected version 1 got 0
I20240501 07:59:12.736740 6166 catalog_manager.cc:5576] TS 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133:39859) does not have the latest schema for tablet 32ecf214d6e14fa3be1b8eafd8c23fac (table rebalancer_dynamic_table [id=95208beb4f5944b09e76c37f268c35ba]). Expected version 1 got 0
I20240501 07:59:12.737066 11521 tablet.cc:1722] T 32ecf214d6e14fa3be1b8eafd8c23fac P a505e7cb133c4392bfda8a4a20874ae9: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 07:59:12.737362 6169 catalog_manager.cc:5576] TS a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607) does not have the latest schema for tablet 32ecf214d6e14fa3be1b8eafd8c23fac (table rebalancer_dynamic_table [id=95208beb4f5944b09e76c37f268c35ba]). Expected version 1 got 0
I20240501 07:59:12.737421 11521 tablet.cc:1620] T 32ecf214d6e14fa3be1b8eafd8c23fac P a505e7cb133c4392bfda8a4a20874ae9: MemRowSet was empty: no flush needed.
I20240501 07:59:12.739851 6169 catalog_manager.cc:5576] TS 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134:40141) does not have the latest schema for tablet 32ecf214d6e14fa3be1b8eafd8c23fac (table rebalancer_dynamic_table [id=95208beb4f5944b09e76c37f268c35ba]). Expected version 1 got 0
I20240501 07:59:12.738173 11522 tablet.cc:1722] T 32ecf214d6e14fa3be1b8eafd8c23fac P 74077b61d6f54342bf9736b4a9f2eade: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 07:59:12.740660 11522 tablet.cc:1620] T 32ecf214d6e14fa3be1b8eafd8c23fac P 74077b61d6f54342bf9736b4a9f2eade: MemRowSet was empty: no flush needed.
I20240501 07:59:12.740713 6171 catalog_manager.cc:5576] TS 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133:39859) does not have the latest schema for tablet 32ecf214d6e14fa3be1b8eafd8c23fac (table rebalancer_dynamic_table [id=95208beb4f5944b09e76c37f268c35ba]). Expected version 1 got 0
I20240501 07:59:12.741158 11523 tablet.cc:1722] T 32ecf214d6e14fa3be1b8eafd8c23fac P 13d05ebdca8c4f489c597ce550dd773d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 07:59:12.741462 11523 tablet.cc:1620] T 32ecf214d6e14fa3be1b8eafd8c23fac P 13d05ebdca8c4f489c597ce550dd773d: MemRowSet was empty: no flush needed.
I20240501 07:59:12.742867 6169 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 0bc706e85ce74cef97a4827ea4d4aa65: rebalancer_dynamic_table [id=95208beb4f5944b09e76c37f268c35ba] alter complete (version 1)
I20240501 07:59:12.743800 9489 raft_consensus.cc:1050] T 782e37a4ebb34a68b461a74115ae8703 P 8315d22c56f7439a9e1fc82f153f018c: attempting to promote NON_VOTER 83f20e84af4c40d09a4804b6b26e697f to VOTER
I20240501 07:59:12.744252 9489 consensus_queue.cc:237] T 782e37a4ebb34a68b461a74115ae8703 P 8315d22c56f7439a9e1fc82f153f018c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false } }
I20240501 07:59:12.745784 9056 raft_consensus.cc:1261] T 782e37a4ebb34a68b461a74115ae8703 P a505e7cb133c4392bfda8a4a20874ae9 [term 1 FOLLOWER]: Refusing update from remote peer 8315d22c56f7439a9e1fc82f153f018c: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 07:59:12.746140 6313 raft_consensus.cc:1261] T 782e37a4ebb34a68b461a74115ae8703 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Refusing update from remote peer 8315d22c56f7439a9e1fc82f153f018c: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 07:59:12.746256 10111 consensus_queue.cc:1035] T 782e37a4ebb34a68b461a74115ae8703 P 8315d22c56f7439a9e1fc82f153f018c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 07:59:12.747074 9189 raft_consensus.cc:1261] T 782e37a4ebb34a68b461a74115ae8703 P 83f20e84af4c40d09a4804b6b26e697f [term 1 LEARNER]: Refusing update from remote peer 8315d22c56f7439a9e1fc82f153f018c: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 07:59:12.747164 9116 consensus_queue.cc:1035] T 782e37a4ebb34a68b461a74115ae8703 P 8315d22c56f7439a9e1fc82f153f018c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 07:59:12.748541 9116 consensus_queue.cc:1035] T 782e37a4ebb34a68b461a74115ae8703 P 8315d22c56f7439a9e1fc82f153f018c [LEADER]: Connected to new peer: Peer: permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 07:59:12.750809 9116 raft_consensus.cc:2932] T 782e37a4ebb34a68b461a74115ae8703 P 8315d22c56f7439a9e1fc82f153f018c [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false } } }
I20240501 07:59:12.751659 6313 raft_consensus.cc:2932] T 782e37a4ebb34a68b461a74115ae8703 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false } } }
I20240501 07:59:12.751765 9189 raft_consensus.cc:2932] T 782e37a4ebb34a68b461a74115ae8703 P 83f20e84af4c40d09a4804b6b26e697f [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false } } }
I20240501 07:59:12.752717 9056 raft_consensus.cc:2932] T 782e37a4ebb34a68b461a74115ae8703 P a505e7cb133c4392bfda8a4a20874ae9 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false } } }
I20240501 07:59:12.754688 6171 catalog_manager.cc:5484] T 782e37a4ebb34a68b461a74115ae8703 P 8315d22c56f7439a9e1fc82f153f018c reported cstate change: config changed from index 7 to 8, 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8315d22c56f7439a9e1fc82f153f018c" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 07:59:12.758638 6444 consensus_queue.cc:237] T 782e37a4ebb34a68b461a74115ae8703 P 8315d22c56f7439a9e1fc82f153f018c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false } }
I20240501 07:59:12.759769 9056 raft_consensus.cc:1261] T 782e37a4ebb34a68b461a74115ae8703 P a505e7cb133c4392bfda8a4a20874ae9 [term 1 FOLLOWER]: Refusing update from remote peer 8315d22c56f7439a9e1fc82f153f018c: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 07:59:12.759816 9189 raft_consensus.cc:1261] T 782e37a4ebb34a68b461a74115ae8703 P 83f20e84af4c40d09a4804b6b26e697f [term 1 FOLLOWER]: Refusing update from remote peer 8315d22c56f7439a9e1fc82f153f018c: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 07:59:12.759922 11346 ts_tablet_manager.cc:902] T df00e37199e944589a0a5f3feeee308a P 13d05ebdca8c4f489c597ce550dd773d: Initiating tablet copy from peer 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130:32941)
I20240501 07:59:12.760510 11346 tablet_copy_client.cc:308] T df00e37199e944589a0a5f3feeee308a P 13d05ebdca8c4f489c597ce550dd773d: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:32941
I20240501 07:59:12.760735 9489 consensus_queue.cc:1035] T 782e37a4ebb34a68b461a74115ae8703 P 8315d22c56f7439a9e1fc82f153f018c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 07:59:12.761162 6466 tablet_copy_service.cc:140] P 8315d22c56f7439a9e1fc82f153f018c: Received BeginTabletCopySession request for tablet df00e37199e944589a0a5f3feeee308a from peer 13d05ebdca8c4f489c597ce550dd773d ({username='slave'} at 127.0.99.134:41693)
I20240501 07:59:12.761258 6466 tablet_copy_service.cc:161] P 8315d22c56f7439a9e1fc82f153f018c: Beginning new tablet copy session on tablet df00e37199e944589a0a5f3feeee308a from peer 13d05ebdca8c4f489c597ce550dd773d at {username='slave'} at 127.0.99.134:41693: session id = 13d05ebdca8c4f489c597ce550dd773d-df00e37199e944589a0a5f3feeee308a
I20240501 07:59:12.761514 10111 consensus_queue.cc:1035] T 782e37a4ebb34a68b461a74115ae8703 P 8315d22c56f7439a9e1fc82f153f018c [LEADER]: Connected to new peer: Peer: permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 07:59:12.762238 6466 tablet_copy_source_session.cc:205] T df00e37199e944589a0a5f3feeee308a P 8315d22c56f7439a9e1fc82f153f018c: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:12.763612 9489 raft_consensus.cc:2932] T 782e37a4ebb34a68b461a74115ae8703 P 8315d22c56f7439a9e1fc82f153f018c [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false } } }
I20240501 07:59:12.763870 11346 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df00e37199e944589a0a5f3feeee308a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:12.765228 6159 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 782e37a4ebb34a68b461a74115ae8703 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 07:59:12.765761 9189 raft_consensus.cc:2932] T 782e37a4ebb34a68b461a74115ae8703 P 83f20e84af4c40d09a4804b6b26e697f [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false } } }
I20240501 07:59:12.765792 9056 raft_consensus.cc:2932] T 782e37a4ebb34a68b461a74115ae8703 P a505e7cb133c4392bfda8a4a20874ae9 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false } } }
I20240501 07:59:12.768203 6169 catalog_manager.cc:5484] T 782e37a4ebb34a68b461a74115ae8703 P 8315d22c56f7439a9e1fc82f153f018c reported cstate change: config changed from index 8 to 9, VOTER 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "8315d22c56f7439a9e1fc82f153f018c" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 07:59:12.770315 11346 tablet_copy_client.cc:783] T df00e37199e944589a0a5f3feeee308a P 13d05ebdca8c4f489c597ce550dd773d: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:12.770480 11346 tablet_copy_client.cc:647] T df00e37199e944589a0a5f3feeee308a P 13d05ebdca8c4f489c597ce550dd773d: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:12.770536 6577 consensus_queue.cc:237] T dc550f54e8024d47b1955905e7858a1e P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: true } }
I20240501 07:59:12.771504 11346 tablet_copy_client.cc:523] T df00e37199e944589a0a5f3feeee308a P 13d05ebdca8c4f489c597ce550dd773d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:12.772238 6444 raft_consensus.cc:1261] T dc550f54e8024d47b1955905e7858a1e P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Refusing update from remote peer 3b164cbbc00f4e8eb3254ddd349a3ff9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:59:12.772349 6707 raft_consensus.cc:1261] T dc550f54e8024d47b1955905e7858a1e P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Refusing update from remote peer 3b164cbbc00f4e8eb3254ddd349a3ff9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:59:12.772727 9507 consensus_queue.cc:1035] T dc550f54e8024d47b1955905e7858a1e P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:12.773074 6295 tablet_service.cc:1514] Processing DeleteTablet for tablet 782e37a4ebb34a68b461a74115ae8703 with delete_type TABLET_DATA_TOMBSTONED (TS 8b8b3ad3aa8f4053b2167dd550f2086f not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:45480
I20240501 07:59:12.773141 9507 consensus_queue.cc:1035] T dc550f54e8024d47b1955905e7858a1e P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:12.773326 10309 tablet_replica.cc:330] T 782e37a4ebb34a68b461a74115ae8703 P 8b8b3ad3aa8f4053b2167dd550f2086f: stopping tablet replica
I20240501 07:59:12.773476 10309 raft_consensus.cc:2229] T 782e37a4ebb34a68b461a74115ae8703 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:12.773659 10309 raft_consensus.cc:2258] T 782e37a4ebb34a68b461a74115ae8703 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Raft consensus is shut down!
W20240501 07:59:12.774165 6510 consensus_peers.cc:485] T dc550f54e8024d47b1955905e7858a1e P 3b164cbbc00f4e8eb3254ddd349a3ff9 -> Peer 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133:39859): Couldn't send request to peer 74077b61d6f54342bf9736b4a9f2eade. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dc550f54e8024d47b1955905e7858a1e. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:12.774629 11346 tablet_bootstrap.cc:492] T df00e37199e944589a0a5f3feeee308a P 13d05ebdca8c4f489c597ce550dd773d: Bootstrap starting.
I20240501 07:59:12.774760 10309 ts_tablet_manager.cc:1875] T 782e37a4ebb34a68b461a74115ae8703 P 8b8b3ad3aa8f4053b2167dd550f2086f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:59:12.777048 10309 ts_tablet_manager.cc:1888] T 782e37a4ebb34a68b461a74115ae8703 P 8b8b3ad3aa8f4053b2167dd550f2086f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240501 07:59:12.777140 10309 log.cc:1196] T 782e37a4ebb34a68b461a74115ae8703 P 8b8b3ad3aa8f4053b2167dd550f2086f: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/782e37a4ebb34a68b461a74115ae8703
I20240501 07:59:12.776950 11119 raft_consensus.cc:2932] T dc550f54e8024d47b1955905e7858a1e P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: true } } }
I20240501 07:59:12.777350 6444 raft_consensus.cc:2932] T dc550f54e8024d47b1955905e7858a1e P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: true } } }
I20240501 07:59:12.778455 6157 catalog_manager.cc:4830] TS 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129:46875): tablet 782e37a4ebb34a68b461a74115ae8703 (table rebalance_test_table_3 [id=eb6765f7643442ff82fde7f790b2aaed]) successfully deleted
I20240501 07:59:12.779907 6707 raft_consensus.cc:2932] T dc550f54e8024d47b1955905e7858a1e P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: true } } }
I20240501 07:59:12.780239 6169 catalog_manager.cc:5484] T dc550f54e8024d47b1955905e7858a1e P 3b164cbbc00f4e8eb3254ddd349a3ff9 reported cstate change: config changed from index -1 to 4, NON_VOTER 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 07:59:12.789435 9056 consensus_queue.cc:237] T f88fe033bce14bf0b1b63a52a7ba137e P a505e7cb133c4392bfda8a4a20874ae9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { promote: true } }
I20240501 07:59:12.790974 9189 raft_consensus.cc:1261] T f88fe033bce14bf0b1b63a52a7ba137e P 83f20e84af4c40d09a4804b6b26e697f [term 1 FOLLOWER]: Refusing update from remote peer a505e7cb133c4392bfda8a4a20874ae9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:12.790974 8924 raft_consensus.cc:1261] T f88fe033bce14bf0b1b63a52a7ba137e P 13d05ebdca8c4f489c597ce550dd773d [term 1 FOLLOWER]: Refusing update from remote peer a505e7cb133c4392bfda8a4a20874ae9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 07:59:12.791384 8991 consensus_peers.cc:485] T f88fe033bce14bf0b1b63a52a7ba137e P a505e7cb133c4392bfda8a4a20874ae9 -> Peer 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130:32941): Couldn't send request to peer 8315d22c56f7439a9e1fc82f153f018c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f88fe033bce14bf0b1b63a52a7ba137e. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:12.791456 10945 consensus_queue.cc:1035] T f88fe033bce14bf0b1b63a52a7ba137e P a505e7cb133c4392bfda8a4a20874ae9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 07:59:12.791689 9940 consensus_queue.cc:1035] T f88fe033bce14bf0b1b63a52a7ba137e P a505e7cb133c4392bfda8a4a20874ae9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 07:59:12.793279 9940 raft_consensus.cc:2932] T f88fe033bce14bf0b1b63a52a7ba137e P a505e7cb133c4392bfda8a4a20874ae9 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { promote: true } } }
I20240501 07:59:12.795651 9189 raft_consensus.cc:2932] T f88fe033bce14bf0b1b63a52a7ba137e P 83f20e84af4c40d09a4804b6b26e697f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { promote: true } } }
I20240501 07:59:12.796800 8924 raft_consensus.cc:2932] T f88fe033bce14bf0b1b63a52a7ba137e P 13d05ebdca8c4f489c597ce550dd773d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { promote: true } } }
I20240501 07:59:12.797914 6169 catalog_manager.cc:5484] T f88fe033bce14bf0b1b63a52a7ba137e P a505e7cb133c4392bfda8a4a20874ae9 reported cstate change: config changed from index -1 to 2, NON_VOTER 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130) added. New cstate: current_term: 1 leader_uuid: "a505e7cb133c4392bfda8a4a20874ae9" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 07:59:12.804234 11346 tablet_bootstrap.cc:492] T df00e37199e944589a0a5f3feeee308a P 13d05ebdca8c4f489c597ce550dd773d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:12.804507 11346 tablet_bootstrap.cc:492] T df00e37199e944589a0a5f3feeee308a P 13d05ebdca8c4f489c597ce550dd773d: Bootstrap complete.
I20240501 07:59:12.804674 11346 ts_tablet_manager.cc:1367] T df00e37199e944589a0a5f3feeee308a P 13d05ebdca8c4f489c597ce550dd773d: Time spent bootstrapping tablet: real 0.030s user 0.020s sys 0.002s
I20240501 07:59:12.805109 11346 raft_consensus.cc:348] T df00e37199e944589a0a5f3feeee308a P 13d05ebdca8c4f489c597ce550dd773d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: true } }
I20240501 07:59:12.805215 11346 raft_consensus.cc:729] T df00e37199e944589a0a5f3feeee308a P 13d05ebdca8c4f489c597ce550dd773d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 13d05ebdca8c4f489c597ce550dd773d, State: Initialized, Role: LEARNER
I20240501 07:59:12.805307 11346 consensus_queue.cc:260] T df00e37199e944589a0a5f3feeee308a P 13d05ebdca8c4f489c597ce550dd773d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: true } }
I20240501 07:59:12.805619 11346 ts_tablet_manager.cc:1398] T df00e37199e944589a0a5f3feeee308a P 13d05ebdca8c4f489c597ce550dd773d: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240501 07:59:12.806277 6466 tablet_copy_service.cc:342] P 8315d22c56f7439a9e1fc82f153f018c: Request end of tablet copy session 13d05ebdca8c4f489c597ce550dd773d-df00e37199e944589a0a5f3feeee308a received from {username='slave'} at 127.0.99.134:41693
I20240501 07:59:12.806377 6466 tablet_copy_service.cc:433] P 8315d22c56f7439a9e1fc82f153f018c: ending tablet copy session 13d05ebdca8c4f489c597ce550dd773d-df00e37199e944589a0a5f3feeee308a on tablet df00e37199e944589a0a5f3feeee308a with peer 13d05ebdca8c4f489c597ce550dd773d
I20240501 07:59:12.828653 8924 raft_consensus.cc:1203] T 8454f24b0c28451b8ee51be39ca3e07f P 13d05ebdca8c4f489c597ce550dd773d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240501 07:59:12.845291 6164 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55950:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 07:59:12.852659 11522 tablet.cc:1722] T 32ecf214d6e14fa3be1b8eafd8c23fac P 74077b61d6f54342bf9736b4a9f2eade: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 07:59:12.853013 11522 tablet.cc:1620] T 32ecf214d6e14fa3be1b8eafd8c23fac P 74077b61d6f54342bf9736b4a9f2eade: MemRowSet was empty: no flush needed.
I20240501 07:59:12.857079 11521 tablet.cc:1722] T 32ecf214d6e14fa3be1b8eafd8c23fac P a505e7cb133c4392bfda8a4a20874ae9: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 07:59:12.857404 11521 tablet.cc:1620] T 32ecf214d6e14fa3be1b8eafd8c23fac P a505e7cb133c4392bfda8a4a20874ae9: MemRowSet was empty: no flush needed.
I20240501 07:59:12.860313 11523 tablet.cc:1722] T 32ecf214d6e14fa3be1b8eafd8c23fac P 13d05ebdca8c4f489c597ce550dd773d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 07:59:12.860600 11523 tablet.cc:1620] T 32ecf214d6e14fa3be1b8eafd8c23fac P 13d05ebdca8c4f489c597ce550dd773d: MemRowSet was empty: no flush needed.
I20240501 07:59:12.861922 6164 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 0bc706e85ce74cef97a4827ea4d4aa65: rebalancer_dynamic_table [id=95208beb4f5944b09e76c37f268c35ba] alter complete (version 2)
I20240501 07:59:12.879148 8924 raft_consensus.cc:1203] T 7dd6566ccb6e43388146a6dfd1c4a88c P 13d05ebdca8c4f489c597ce550dd773d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 07:59:12.927794 8905 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 07:59:12.929358 6556 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 07:59:12.929901 6687 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 07:59:12.930603 6424 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 07:59:12.935489 10478 consensus_queue.cc:1035] T 0864c290fafa488abefa93d41c0a2efe P 13d05ebdca8c4f489c597ce550dd773d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:12.940276 11346 ts_tablet_manager.cc:902] T 8663a511521644009fc360ad92c093a1 P 13d05ebdca8c4f489c597ce550dd773d: Initiating tablet copy from peer 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132:36773)
I20240501 07:59:12.940719 11346 tablet_copy_client.cc:308] T 8663a511521644009fc360ad92c093a1 P 13d05ebdca8c4f489c597ce550dd773d: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:36773
I20240501 07:59:12.940706 9035 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 07:59:12.941742 10235 consensus_queue.cc:1035] T 0864c290fafa488abefa93d41c0a2efe P 13d05ebdca8c4f489c597ce550dd773d [LEADER]: Connected to new peer: Peer: permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:12.944762 6728 tablet_copy_service.cc:140] P 1f9cb72be98941b5a8e504cbe34132b9: Received BeginTabletCopySession request for tablet 8663a511521644009fc360ad92c093a1 from peer 13d05ebdca8c4f489c597ce550dd773d ({username='slave'} at 127.0.99.134:44489)
I20240501 07:59:12.944878 6728 tablet_copy_service.cc:161] P 1f9cb72be98941b5a8e504cbe34132b9: Beginning new tablet copy session on tablet 8663a511521644009fc360ad92c093a1 from peer 13d05ebdca8c4f489c597ce550dd773d at {username='slave'} at 127.0.99.134:44489: session id = 13d05ebdca8c4f489c597ce550dd773d-8663a511521644009fc360ad92c093a1
I20240501 07:59:12.945729 9169 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 07:59:12.946069 6728 tablet_copy_source_session.cc:205] T 8663a511521644009fc360ad92c093a1 P 1f9cb72be98941b5a8e504cbe34132b9: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:12.947012 8757 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 07:59:12.953595 11346 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8663a511521644009fc360ad92c093a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:12.956060 11346 tablet_copy_client.cc:783] T 8663a511521644009fc360ad92c093a1 P 13d05ebdca8c4f489c597ce550dd773d: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:12.956243 11346 tablet_copy_client.cc:647] T 8663a511521644009fc360ad92c093a1 P 13d05ebdca8c4f489c597ce550dd773d: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:12.963687 6169 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55950:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 07:59:12.967038 11346 tablet_copy_client.cc:523] T 8663a511521644009fc360ad92c093a1 P 13d05ebdca8c4f489c597ce550dd773d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:12.968900 11346 tablet_bootstrap.cc:492] T 8663a511521644009fc360ad92c093a1 P 13d05ebdca8c4f489c597ce550dd773d: Bootstrap starting.
I20240501 07:59:12.973830 11521 tablet.cc:1722] T 32ecf214d6e14fa3be1b8eafd8c23fac P a505e7cb133c4392bfda8a4a20874ae9: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 07:59:12.975657 11522 tablet.cc:1722] T 32ecf214d6e14fa3be1b8eafd8c23fac P 74077b61d6f54342bf9736b4a9f2eade: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 07:59:12.977568 11611 ts_tablet_manager.cc:902] T 239ee596e62d47169eeb6670dc62d59c P 1f9cb72be98941b5a8e504cbe34132b9: Initiating tablet copy from peer 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134:40141)
I20240501 07:59:12.979409 11107 raft_consensus.cc:1050] T 6e91683ff22243c1b3877733d74e2886 P 3b164cbbc00f4e8eb3254ddd349a3ff9: attempting to promote NON_VOTER 74077b61d6f54342bf9736b4a9f2eade to VOTER
I20240501 07:59:12.979813 11107 consensus_queue.cc:237] T 6e91683ff22243c1b3877733d74e2886 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } }
I20240501 07:59:12.981269 6313 raft_consensus.cc:1261] T 6e91683ff22243c1b3877733d74e2886 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Refusing update from remote peer 3b164cbbc00f4e8eb3254ddd349a3ff9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:59:12.981730 8793 raft_consensus.cc:1261] T 6e91683ff22243c1b3877733d74e2886 P 74077b61d6f54342bf9736b4a9f2eade [term 1 LEARNER]: Refusing update from remote peer 3b164cbbc00f4e8eb3254ddd349a3ff9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:59:12.981809 9507 consensus_queue.cc:1035] T 6e91683ff22243c1b3877733d74e2886 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:59:12.982121 11119 consensus_queue.cc:1035] T 6e91683ff22243c1b3877733d74e2886 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:59:12.982530 6169 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 0bc706e85ce74cef97a4827ea4d4aa65: rebalancer_dynamic_table_new_name [id=95208beb4f5944b09e76c37f268c35ba] alter complete (version 3)
I20240501 07:59:12.983455 6707 raft_consensus.cc:1261] T 6e91683ff22243c1b3877733d74e2886 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Refusing update from remote peer 3b164cbbc00f4e8eb3254ddd349a3ff9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:59:12.983961 11119 consensus_queue.cc:1035] T 6e91683ff22243c1b3877733d74e2886 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:59:12.974574 11523 tablet.cc:1722] T 32ecf214d6e14fa3be1b8eafd8c23fac P 13d05ebdca8c4f489c597ce550dd773d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 07:59:12.985853 11611 tablet_copy_client.cc:308] T 239ee596e62d47169eeb6670dc62d59c P 1f9cb72be98941b5a8e504cbe34132b9: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.134:40141
I20240501 07:59:12.987066 8945 tablet_copy_service.cc:140] P 13d05ebdca8c4f489c597ce550dd773d: Received BeginTabletCopySession request for tablet 239ee596e62d47169eeb6670dc62d59c from peer 1f9cb72be98941b5a8e504cbe34132b9 ({username='slave'} at 127.0.99.132:43109)
I20240501 07:59:12.987160 8945 tablet_copy_service.cc:161] P 13d05ebdca8c4f489c597ce550dd773d: Beginning new tablet copy session on tablet 239ee596e62d47169eeb6670dc62d59c from peer 1f9cb72be98941b5a8e504cbe34132b9 at {username='slave'} at 127.0.99.132:43109: session id = 1f9cb72be98941b5a8e504cbe34132b9-239ee596e62d47169eeb6670dc62d59c
I20240501 07:59:12.987881 8945 tablet_copy_source_session.cc:205] T 239ee596e62d47169eeb6670dc62d59c P 13d05ebdca8c4f489c597ce550dd773d: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:12.989423 11611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 239ee596e62d47169eeb6670dc62d59c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:12.991739 11611 tablet_copy_client.cc:783] T 239ee596e62d47169eeb6670dc62d59c P 1f9cb72be98941b5a8e504cbe34132b9: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:12.991902 11611 tablet_copy_client.cc:647] T 239ee596e62d47169eeb6670dc62d59c P 1f9cb72be98941b5a8e504cbe34132b9: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:12.995697 6295 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 07:59:12.995978 11611 tablet_copy_client.cc:523] T 239ee596e62d47169eeb6670dc62d59c P 1f9cb72be98941b5a8e504cbe34132b9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:12.996255 8680 raft_consensus.cc:2932] T 6e91683ff22243c1b3877733d74e2886 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } } }
I20240501 07:59:12.997684 11611 tablet_bootstrap.cc:492] T 239ee596e62d47169eeb6670dc62d59c P 1f9cb72be98941b5a8e504cbe34132b9: Bootstrap starting.
I20240501 07:59:12.998270 6313 raft_consensus.cc:2932] T 6e91683ff22243c1b3877733d74e2886 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } } }
I20240501 07:59:12.998812 6707 raft_consensus.cc:2932] T 6e91683ff22243c1b3877733d74e2886 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } } }
I20240501 07:59:13.002166 6171 catalog_manager.cc:5484] T 6e91683ff22243c1b3877733d74e2886 P 8b8b3ad3aa8f4053b2167dd550f2086f reported cstate change: config changed from index 4 to 5, 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } } }
I20240501 07:59:13.006045 11611 tablet_bootstrap.cc:492] T 239ee596e62d47169eeb6670dc62d59c P 1f9cb72be98941b5a8e504cbe34132b9: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:13.006326 11611 tablet_bootstrap.cc:492] T 239ee596e62d47169eeb6670dc62d59c P 1f9cb72be98941b5a8e504cbe34132b9: Bootstrap complete.
I20240501 07:59:13.006507 11611 ts_tablet_manager.cc:1367] T 239ee596e62d47169eeb6670dc62d59c P 1f9cb72be98941b5a8e504cbe34132b9: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.000s
I20240501 07:59:13.006994 11611 raft_consensus.cc:348] T 239ee596e62d47169eeb6670dc62d59c P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { promote: true } }
I20240501 07:59:13.007112 11611 raft_consensus.cc:729] T 239ee596e62d47169eeb6670dc62d59c P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1f9cb72be98941b5a8e504cbe34132b9, State: Initialized, Role: LEARNER
I20240501 07:59:13.007185 11611 consensus_queue.cc:260] T 239ee596e62d47169eeb6670dc62d59c P 1f9cb72be98941b5a8e504cbe34132b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { promote: true } }
I20240501 07:59:13.007462 11611 ts_tablet_manager.cc:1398] T 239ee596e62d47169eeb6670dc62d59c P 1f9cb72be98941b5a8e504cbe34132b9: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:59:13.008561 8945 tablet_copy_service.cc:342] P 13d05ebdca8c4f489c597ce550dd773d: Request end of tablet copy session 1f9cb72be98941b5a8e504cbe34132b9-239ee596e62d47169eeb6670dc62d59c received from {username='slave'} at 127.0.99.132:43109
I20240501 07:59:13.008648 8945 tablet_copy_service.cc:433] P 13d05ebdca8c4f489c597ce550dd773d: ending tablet copy session 1f9cb72be98941b5a8e504cbe34132b9-239ee596e62d47169eeb6670dc62d59c on tablet 239ee596e62d47169eeb6670dc62d59c with peer 1f9cb72be98941b5a8e504cbe34132b9
I20240501 07:59:13.012306 11317 ts_tablet_manager.cc:902] T a756205204d146ce82a2799e30582f57 P a505e7cb133c4392bfda8a4a20874ae9: Initiating tablet copy from peer 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130:32941)
I20240501 07:59:13.012773 11317 tablet_copy_client.cc:308] T a756205204d146ce82a2799e30582f57 P a505e7cb133c4392bfda8a4a20874ae9: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:32941
I20240501 07:59:13.013702 6466 tablet_copy_service.cc:140] P 8315d22c56f7439a9e1fc82f153f018c: Received BeginTabletCopySession request for tablet a756205204d146ce82a2799e30582f57 from peer a505e7cb133c4392bfda8a4a20874ae9 ({username='slave'} at 127.0.99.135:36631)
I20240501 07:59:13.013808 6466 tablet_copy_service.cc:161] P 8315d22c56f7439a9e1fc82f153f018c: Beginning new tablet copy session on tablet a756205204d146ce82a2799e30582f57 from peer a505e7cb133c4392bfda8a4a20874ae9 at {username='slave'} at 127.0.99.135:36631: session id = a505e7cb133c4392bfda8a4a20874ae9-a756205204d146ce82a2799e30582f57
I20240501 07:59:13.014880 6466 tablet_copy_source_session.cc:205] T a756205204d146ce82a2799e30582f57 P 8315d22c56f7439a9e1fc82f153f018c: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:13.015857 9056 raft_consensus.cc:1203] T d3f2250a62c6429e886753aaffcbe883 P a505e7cb133c4392bfda8a4a20874ae9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240501 07:59:13.018383 11317 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a756205204d146ce82a2799e30582f57. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:13.020483 11317 tablet_copy_client.cc:783] T a756205204d146ce82a2799e30582f57 P a505e7cb133c4392bfda8a4a20874ae9: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:13.021020 11317 tablet_copy_client.cc:647] T a756205204d146ce82a2799e30582f57 P a505e7cb133c4392bfda8a4a20874ae9: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:13.023459 11317 tablet_copy_client.cc:523] T a756205204d146ce82a2799e30582f57 P a505e7cb133c4392bfda8a4a20874ae9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:13.024592 11317 tablet_bootstrap.cc:492] T a756205204d146ce82a2799e30582f57 P a505e7cb133c4392bfda8a4a20874ae9: Bootstrap starting.
I20240501 07:59:13.022059 8793 raft_consensus.cc:2932] T 6e91683ff22243c1b3877733d74e2886 P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } } }
I20240501 07:59:13.026950 10235 consensus_queue.cc:1035] T 59e171892fc1414fbcfd6a8c090854f3 P 13d05ebdca8c4f489c597ce550dd773d [LEADER]: Connected to new peer: Peer: permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:13.031241 10478 consensus_queue.cc:1035] T 59e171892fc1414fbcfd6a8c090854f3 P 13d05ebdca8c4f489c597ce550dd773d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:13.032189 11109 raft_consensus.cc:1050] T 2976d6353e8947b7b6c37f72efbeeac4 P 83f20e84af4c40d09a4804b6b26e697f: attempting to promote NON_VOTER 3b164cbbc00f4e8eb3254ddd349a3ff9 to VOTER
I20240501 07:59:13.032656 11109 consensus_queue.cc:237] T 2976d6353e8947b7b6c37f72efbeeac4 P 83f20e84af4c40d09a4804b6b26e697f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: false } }
I20240501 07:59:13.034430 9056 raft_consensus.cc:1261] T 2976d6353e8947b7b6c37f72efbeeac4 P a505e7cb133c4392bfda8a4a20874ae9 [term 1 FOLLOWER]: Refusing update from remote peer 83f20e84af4c40d09a4804b6b26e697f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 07:59:13.034994 11109 consensus_queue.cc:1035] T 2976d6353e8947b7b6c37f72efbeeac4 P 83f20e84af4c40d09a4804b6b26e697f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 07:59:13.036036 6313 raft_consensus.cc:1261] T 2976d6353e8947b7b6c37f72efbeeac4 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Refusing update from remote peer 83f20e84af4c40d09a4804b6b26e697f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 07:59:13.036556 6575 raft_consensus.cc:1261] T 2976d6353e8947b7b6c37f72efbeeac4 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 LEARNER]: Refusing update from remote peer 83f20e84af4c40d09a4804b6b26e697f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 07:59:13.038331 11109 consensus_queue.cc:1035] T 2976d6353e8947b7b6c37f72efbeeac4 P 83f20e84af4c40d09a4804b6b26e697f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 07:59:13.038823 11109 consensus_queue.cc:1035] T 2976d6353e8947b7b6c37f72efbeeac4 P 83f20e84af4c40d09a4804b6b26e697f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 07:59:13.040549 11630 raft_consensus.cc:2932] T 2976d6353e8947b7b6c37f72efbeeac4 P 83f20e84af4c40d09a4804b6b26e697f [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: false } } }
I20240501 07:59:13.040917 6577 raft_consensus.cc:2932] T 2976d6353e8947b7b6c37f72efbeeac4 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: false } } }
I20240501 07:59:13.043138 9056 raft_consensus.cc:2932] T 2976d6353e8947b7b6c37f72efbeeac4 P a505e7cb133c4392bfda8a4a20874ae9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: false } } }
I20240501 07:59:13.043766 6166 catalog_manager.cc:5484] T 2976d6353e8947b7b6c37f72efbeeac4 P 83f20e84af4c40d09a4804b6b26e697f reported cstate change: config changed from index 2 to 3, 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "83f20e84af4c40d09a4804b6b26e697f" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 07:59:13.057621 11317 tablet_bootstrap.cc:492] T a756205204d146ce82a2799e30582f57 P a505e7cb133c4392bfda8a4a20874ae9: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:13.057966 11317 tablet_bootstrap.cc:492] T a756205204d146ce82a2799e30582f57 P a505e7cb133c4392bfda8a4a20874ae9: Bootstrap complete.
I20240501 07:59:13.058172 11317 ts_tablet_manager.cc:1367] T a756205204d146ce82a2799e30582f57 P a505e7cb133c4392bfda8a4a20874ae9: Time spent bootstrapping tablet: real 0.034s user 0.023s sys 0.002s
I20240501 07:59:13.058230 9189 consensus_queue.cc:237] T 2976d6353e8947b7b6c37f72efbeeac4 P 83f20e84af4c40d09a4804b6b26e697f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: false } }
I20240501 07:59:13.058676 11317 raft_consensus.cc:348] T a756205204d146ce82a2799e30582f57 P a505e7cb133c4392bfda8a4a20874ae9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: true } }
I20240501 07:59:13.058792 11317 raft_consensus.cc:729] T a756205204d146ce82a2799e30582f57 P a505e7cb133c4392bfda8a4a20874ae9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a505e7cb133c4392bfda8a4a20874ae9, State: Initialized, Role: LEARNER
I20240501 07:59:13.058892 11317 consensus_queue.cc:260] T a756205204d146ce82a2799e30582f57 P a505e7cb133c4392bfda8a4a20874ae9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: true } }
I20240501 07:59:13.059260 11317 ts_tablet_manager.cc:1398] T a756205204d146ce82a2799e30582f57 P a505e7cb133c4392bfda8a4a20874ae9: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240501 07:59:13.059566 6577 raft_consensus.cc:1261] T 2976d6353e8947b7b6c37f72efbeeac4 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Refusing update from remote peer 83f20e84af4c40d09a4804b6b26e697f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:59:13.059731 6466 tablet_copy_service.cc:342] P 8315d22c56f7439a9e1fc82f153f018c: Request end of tablet copy session a505e7cb133c4392bfda8a4a20874ae9-a756205204d146ce82a2799e30582f57 received from {username='slave'} at 127.0.99.135:36631
I20240501 07:59:13.059821 6466 tablet_copy_service.cc:433] P 8315d22c56f7439a9e1fc82f153f018c: ending tablet copy session a505e7cb133c4392bfda8a4a20874ae9-a756205204d146ce82a2799e30582f57 on tablet a756205204d146ce82a2799e30582f57 with peer a505e7cb133c4392bfda8a4a20874ae9
I20240501 07:59:13.060448 11109 consensus_queue.cc:1035] T 2976d6353e8947b7b6c37f72efbeeac4 P 83f20e84af4c40d09a4804b6b26e697f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:13.061260 9056 raft_consensus.cc:1261] T 2976d6353e8947b7b6c37f72efbeeac4 P a505e7cb133c4392bfda8a4a20874ae9 [term 1 FOLLOWER]: Refusing update from remote peer 83f20e84af4c40d09a4804b6b26e697f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:59:13.061677 11109 consensus_queue.cc:1035] T 2976d6353e8947b7b6c37f72efbeeac4 P 83f20e84af4c40d09a4804b6b26e697f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:13.063072 11629 raft_consensus.cc:2932] T 2976d6353e8947b7b6c37f72efbeeac4 P 83f20e84af4c40d09a4804b6b26e697f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: false } } }
I20240501 07:59:13.063474 9056 raft_consensus.cc:2932] T 2976d6353e8947b7b6c37f72efbeeac4 P a505e7cb133c4392bfda8a4a20874ae9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: false } } }
I20240501 07:59:13.067271 6158 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 2976d6353e8947b7b6c37f72efbeeac4 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 07:59:13.067905 6577 raft_consensus.cc:2932] T 2976d6353e8947b7b6c37f72efbeeac4 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: false } } }
I20240501 07:59:13.070125 6171 catalog_manager.cc:5484] T 2976d6353e8947b7b6c37f72efbeeac4 P a505e7cb133c4392bfda8a4a20874ae9 reported cstate change: config changed from index 3 to 4, VOTER 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "83f20e84af4c40d09a4804b6b26e697f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: false } } }
I20240501 07:59:13.073289 6295 tablet_service.cc:1514] Processing DeleteTablet for tablet 2976d6353e8947b7b6c37f72efbeeac4 with delete_type TABLET_DATA_TOMBSTONED (TS 8b8b3ad3aa8f4053b2167dd550f2086f not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:45480
I20240501 07:59:13.073511 10309 tablet_replica.cc:330] T 2976d6353e8947b7b6c37f72efbeeac4 P 8b8b3ad3aa8f4053b2167dd550f2086f: stopping tablet replica
I20240501 07:59:13.073644 10309 raft_consensus.cc:2229] T 2976d6353e8947b7b6c37f72efbeeac4 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:13.073768 10309 pending_rounds.cc:62] T 2976d6353e8947b7b6c37f72efbeeac4 P 8b8b3ad3aa8f4053b2167dd550f2086f: Trying to abort 1 pending ops.
I20240501 07:59:13.073794 10309 pending_rounds.cc:68] T 2976d6353e8947b7b6c37f72efbeeac4 P 8b8b3ad3aa8f4053b2167dd550f2086f: Aborting op as it isn't in flight: id { term: 1 index: 3 } timestamp: 7022798246019735552 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "2976d6353e8947b7b6c37f72efbeeac4" old_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: true } } } new_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: false } } } }
I20240501 07:59:13.073917 10309 raft_consensus.cc:2894] T 2976d6353e8947b7b6c37f72efbeeac4 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Aborting config change with OpId 1.3: Aborted: Op aborted
I20240501 07:59:13.073971 10309 raft_consensus.cc:2866] T 2976d6353e8947b7b6c37f72efbeeac4 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240501 07:59:13.074009 10309 raft_consensus.cc:2258] T 2976d6353e8947b7b6c37f72efbeeac4 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:13.074841 10309 ts_tablet_manager.cc:1875] T 2976d6353e8947b7b6c37f72efbeeac4 P 8b8b3ad3aa8f4053b2167dd550f2086f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:59:13.077080 10309 ts_tablet_manager.cc:1888] T 2976d6353e8947b7b6c37f72efbeeac4 P 8b8b3ad3aa8f4053b2167dd550f2086f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 07:59:13.077147 10309 log.cc:1196] T 2976d6353e8947b7b6c37f72efbeeac4 P 8b8b3ad3aa8f4053b2167dd550f2086f: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/2976d6353e8947b7b6c37f72efbeeac4
I20240501 07:59:13.077728 6157 catalog_manager.cc:4830] TS 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129:46875): tablet 2976d6353e8947b7b6c37f72efbeeac4 (table rebalancer_extra_table_6 [id=8b10e9a527644d40b1515561fc5df5c2]) successfully deleted
I20240501 07:59:13.088853 6171 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55950:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 07:59:13.089033 6171 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55950:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 07:59:13.089190 11346 tablet_bootstrap.cc:492] T 8663a511521644009fc360ad92c093a1 P 13d05ebdca8c4f489c597ce550dd773d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:13.089509 11346 tablet_bootstrap.cc:492] T 8663a511521644009fc360ad92c093a1 P 13d05ebdca8c4f489c597ce550dd773d: Bootstrap complete.
I20240501 07:59:13.089709 11346 ts_tablet_manager.cc:1367] T 8663a511521644009fc360ad92c093a1 P 13d05ebdca8c4f489c597ce550dd773d: Time spent bootstrapping tablet: real 0.121s user 0.015s sys 0.008s
I20240501 07:59:13.090662 11346 raft_consensus.cc:348] T 8663a511521644009fc360ad92c093a1 P 13d05ebdca8c4f489c597ce550dd773d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: true } }
I20240501 07:59:13.090991 11346 raft_consensus.cc:729] T 8663a511521644009fc360ad92c093a1 P 13d05ebdca8c4f489c597ce550dd773d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 13d05ebdca8c4f489c597ce550dd773d, State: Initialized, Role: LEARNER
I20240501 07:59:13.091123 11346 consensus_queue.cc:260] T 8663a511521644009fc360ad92c093a1 P 13d05ebdca8c4f489c597ce550dd773d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: true } }
I20240501 07:59:13.091524 11346 ts_tablet_manager.cc:1398] T 8663a511521644009fc360ad92c093a1 P 13d05ebdca8c4f489c597ce550dd773d: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 07:59:13.092114 6728 tablet_copy_service.cc:342] P 1f9cb72be98941b5a8e504cbe34132b9: Request end of tablet copy session 13d05ebdca8c4f489c597ce550dd773d-8663a511521644009fc360ad92c093a1 received from {username='slave'} at 127.0.99.134:44489
I20240501 07:59:13.092195 6728 tablet_copy_service.cc:433] P 1f9cb72be98941b5a8e504cbe34132b9: ending tablet copy session 13d05ebdca8c4f489c597ce550dd773d-8663a511521644009fc360ad92c093a1 on tablet 8663a511521644009fc360ad92c093a1 with peer 13d05ebdca8c4f489c597ce550dd773d
I20240501 07:59:13.093327 6171 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 0bc706e85ce74cef97a4827ea4d4aa65: Sending DeleteTablet for 3 replicas of tablet 32ecf214d6e14fa3be1b8eafd8c23fac
I20240501 07:59:13.094038 8757 tablet_service.cc:1514] Processing DeleteTablet for tablet 32ecf214d6e14fa3be1b8eafd8c23fac with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:13 UTC) from {username='slave'} at 127.0.0.1:43136
I20240501 07:59:13.094043 9035 tablet_service.cc:1514] Processing DeleteTablet for tablet 32ecf214d6e14fa3be1b8eafd8c23fac with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:13 UTC) from {username='slave'} at 127.0.0.1:44236
I20240501 07:59:13.094462 8905 tablet_service.cc:1514] Processing DeleteTablet for tablet 32ecf214d6e14fa3be1b8eafd8c23fac with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:13 UTC) from {username='slave'} at 127.0.0.1:47202
I20240501 07:59:13.095656 11641 tablet_replica.cc:330] T 32ecf214d6e14fa3be1b8eafd8c23fac P a505e7cb133c4392bfda8a4a20874ae9: stopping tablet replica
I20240501 07:59:13.095907 11641 raft_consensus.cc:2229] T 32ecf214d6e14fa3be1b8eafd8c23fac P a505e7cb133c4392bfda8a4a20874ae9 [term 1 LEADER]: Raft consensus shutting down.
I20240501 07:59:13.096309 10324 consensus_queue.cc:1035] T 6daebcfc41754ec4aca939ca20588854 P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:13.097698 11640 tablet_replica.cc:330] T 32ecf214d6e14fa3be1b8eafd8c23fac P 74077b61d6f54342bf9736b4a9f2eade: stopping tablet replica
I20240501 07:59:13.098094 11641 raft_consensus.cc:2258] T 32ecf214d6e14fa3be1b8eafd8c23fac P a505e7cb133c4392bfda8a4a20874ae9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:13.098855 11641 ts_tablet_manager.cc:1875] T 32ecf214d6e14fa3be1b8eafd8c23fac P a505e7cb133c4392bfda8a4a20874ae9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:59:13.100311 11640 raft_consensus.cc:2229] T 32ecf214d6e14fa3be1b8eafd8c23fac P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:13.100517 11640 raft_consensus.cc:2258] T 32ecf214d6e14fa3be1b8eafd8c23fac P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:13.101264 11640 ts_tablet_manager.cc:1875] T 32ecf214d6e14fa3be1b8eafd8c23fac P 74077b61d6f54342bf9736b4a9f2eade: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:59:13.102308 11641 ts_tablet_manager.cc:1888] T 32ecf214d6e14fa3be1b8eafd8c23fac P a505e7cb133c4392bfda8a4a20874ae9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:59:13.102388 11641 log.cc:1196] T 32ecf214d6e14fa3be1b8eafd8c23fac P a505e7cb133c4392bfda8a4a20874ae9: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/32ecf214d6e14fa3be1b8eafd8c23fac
I20240501 07:59:13.102659 11641 ts_tablet_manager.cc:1909] T 32ecf214d6e14fa3be1b8eafd8c23fac P a505e7cb133c4392bfda8a4a20874ae9: Deleting consensus metadata
I20240501 07:59:13.103554 6158 catalog_manager.cc:4830] TS a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): tablet 32ecf214d6e14fa3be1b8eafd8c23fac (table rebalancer_dynamic_table_new_name [id=95208beb4f5944b09e76c37f268c35ba]) successfully deleted
I20240501 07:59:13.104924 11640 ts_tablet_manager.cc:1888] T 32ecf214d6e14fa3be1b8eafd8c23fac P 74077b61d6f54342bf9736b4a9f2eade: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:59:13.105026 11640 log.cc:1196] T 32ecf214d6e14fa3be1b8eafd8c23fac P 74077b61d6f54342bf9736b4a9f2eade: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/32ecf214d6e14fa3be1b8eafd8c23fac
I20240501 07:59:13.105296 11640 ts_tablet_manager.cc:1909] T 32ecf214d6e14fa3be1b8eafd8c23fac P 74077b61d6f54342bf9736b4a9f2eade: Deleting consensus metadata
I20240501 07:59:13.105990 10324 consensus_queue.cc:1035] T 6daebcfc41754ec4aca939ca20588854 P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:13.107724 11642 tablet_replica.cc:330] T 32ecf214d6e14fa3be1b8eafd8c23fac P 13d05ebdca8c4f489c597ce550dd773d: stopping tablet replica
I20240501 07:59:13.107801 6157 catalog_manager.cc:4830] TS 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133:39859): tablet 32ecf214d6e14fa3be1b8eafd8c23fac (table rebalancer_dynamic_table_new_name [id=95208beb4f5944b09e76c37f268c35ba]) successfully deleted
I20240501 07:59:13.107975 11642 raft_consensus.cc:2229] T 32ecf214d6e14fa3be1b8eafd8c23fac P 13d05ebdca8c4f489c597ce550dd773d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:13.108166 11642 raft_consensus.cc:2258] T 32ecf214d6e14fa3be1b8eafd8c23fac P 13d05ebdca8c4f489c597ce550dd773d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:13.108812 11642 ts_tablet_manager.cc:1875] T 32ecf214d6e14fa3be1b8eafd8c23fac P 13d05ebdca8c4f489c597ce550dd773d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:59:13.110843 11642 ts_tablet_manager.cc:1888] T 32ecf214d6e14fa3be1b8eafd8c23fac P 13d05ebdca8c4f489c597ce550dd773d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:59:13.110982 11642 log.cc:1196] T 32ecf214d6e14fa3be1b8eafd8c23fac P 13d05ebdca8c4f489c597ce550dd773d: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/32ecf214d6e14fa3be1b8eafd8c23fac
I20240501 07:59:13.111265 11642 ts_tablet_manager.cc:1909] T 32ecf214d6e14fa3be1b8eafd8c23fac P 13d05ebdca8c4f489c597ce550dd773d: Deleting consensus metadata
I20240501 07:59:13.111918 6159 catalog_manager.cc:4830] TS 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134:40141): tablet 32ecf214d6e14fa3be1b8eafd8c23fac (table rebalancer_dynamic_table_new_name [id=95208beb4f5944b09e76c37f268c35ba]) successfully deleted
I20240501 07:59:13.112207 6169 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55950:
name: "rebalancer_extra_table_14"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 07:59:13.119550 6687 tablet_service.cc:1467] Processing CreateTablet for tablet 2560b1ec81ac4357b32c7a8ba9d72216 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=4df593f62ca14b82ae95b3b691699a61]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:13.119874 6687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2560b1ec81ac4357b32c7a8ba9d72216. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:13.121438 9169 tablet_service.cc:1467] Processing CreateTablet for tablet e0583fa43994452ca7553cbf2309eba1 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=4df593f62ca14b82ae95b3b691699a61]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:13.121562 8905 tablet_service.cc:1467] Processing CreateTablet for tablet e0583fa43994452ca7553cbf2309eba1 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=4df593f62ca14b82ae95b3b691699a61]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:13.121759 9169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0583fa43994452ca7553cbf2309eba1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:13.121810 8905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0583fa43994452ca7553cbf2309eba1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:13.122227 11652 tablet_bootstrap.cc:492] T 2560b1ec81ac4357b32c7a8ba9d72216 P 1f9cb72be98941b5a8e504cbe34132b9: Bootstrap starting.
I20240501 07:59:13.123157 11652 tablet_bootstrap.cc:654] T 2560b1ec81ac4357b32c7a8ba9d72216 P 1f9cb72be98941b5a8e504cbe34132b9: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:13.124305 9035 tablet_service.cc:1467] Processing CreateTablet for tablet e5bd3a8f7a7640c09774f781009eb19c (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=4df593f62ca14b82ae95b3b691699a61]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:13.124372 9036 tablet_service.cc:1467] Processing CreateTablet for tablet e0583fa43994452ca7553cbf2309eba1 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=4df593f62ca14b82ae95b3b691699a61]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:13.124578 9035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5bd3a8f7a7640c09774f781009eb19c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:13.124697 9036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0583fa43994452ca7553cbf2309eba1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:13.125011 9034 tablet_service.cc:1467] Processing CreateTablet for tablet 2560b1ec81ac4357b32c7a8ba9d72216 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=4df593f62ca14b82ae95b3b691699a61]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:13.125221 9034 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2560b1ec81ac4357b32c7a8ba9d72216. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:13.125170 6295 tablet_service.cc:1467] Processing CreateTablet for tablet e5bd3a8f7a7640c09774f781009eb19c (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=4df593f62ca14b82ae95b3b691699a61]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:13.125430 6295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5bd3a8f7a7640c09774f781009eb19c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:13.127120 11656 tablet_bootstrap.cc:492] T e0583fa43994452ca7553cbf2309eba1 P a505e7cb133c4392bfda8a4a20874ae9: Bootstrap starting.
I20240501 07:59:13.127218 8774 tablet_service.cc:1467] Processing CreateTablet for tablet 2560b1ec81ac4357b32c7a8ba9d72216 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=4df593f62ca14b82ae95b3b691699a61]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:13.127490 8774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2560b1ec81ac4357b32c7a8ba9d72216. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:13.127785 11652 tablet_bootstrap.cc:492] T 2560b1ec81ac4357b32c7a8ba9d72216 P 1f9cb72be98941b5a8e504cbe34132b9: No bootstrap required, opened a new log
I20240501 07:59:13.127888 11652 ts_tablet_manager.cc:1367] T 2560b1ec81ac4357b32c7a8ba9d72216 P 1f9cb72be98941b5a8e504cbe34132b9: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 07:59:13.129014 11656 tablet_bootstrap.cc:654] T e0583fa43994452ca7553cbf2309eba1 P a505e7cb133c4392bfda8a4a20874ae9: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:13.128932 11652 raft_consensus.cc:348] T 2560b1ec81ac4357b32c7a8ba9d72216 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } }
I20240501 07:59:13.129074 11652 raft_consensus.cc:374] T 2560b1ec81ac4357b32c7a8ba9d72216 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:13.129109 11652 raft_consensus.cc:729] T 2560b1ec81ac4357b32c7a8ba9d72216 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f9cb72be98941b5a8e504cbe34132b9, State: Initialized, Role: FOLLOWER
I20240501 07:59:13.129284 11652 consensus_queue.cc:260] T 2560b1ec81ac4357b32c7a8ba9d72216 P 1f9cb72be98941b5a8e504cbe34132b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } }
I20240501 07:59:13.129580 11652 ts_tablet_manager.cc:1398] T 2560b1ec81ac4357b32c7a8ba9d72216 P 1f9cb72be98941b5a8e504cbe34132b9: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:59:13.130028 11655 tablet_bootstrap.cc:492] T e0583fa43994452ca7553cbf2309eba1 P 13d05ebdca8c4f489c597ce550dd773d: Bootstrap starting.
I20240501 07:59:13.130368 11656 tablet_bootstrap.cc:492] T e0583fa43994452ca7553cbf2309eba1 P a505e7cb133c4392bfda8a4a20874ae9: No bootstrap required, opened a new log
I20240501 07:59:13.130458 11656 ts_tablet_manager.cc:1367] T e0583fa43994452ca7553cbf2309eba1 P a505e7cb133c4392bfda8a4a20874ae9: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.002s
I20240501 07:59:13.127193 8757 tablet_service.cc:1467] Processing CreateTablet for tablet e5bd3a8f7a7640c09774f781009eb19c (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=4df593f62ca14b82ae95b3b691699a61]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:13.130817 8757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5bd3a8f7a7640c09774f781009eb19c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:13.130896 11656 raft_consensus.cc:348] T e0583fa43994452ca7553cbf2309eba1 P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } }
I20240501 07:59:13.131026 11656 raft_consensus.cc:374] T e0583fa43994452ca7553cbf2309eba1 P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:13.131034 11655 tablet_bootstrap.cc:654] T e0583fa43994452ca7553cbf2309eba1 P 13d05ebdca8c4f489c597ce550dd773d: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:13.131065 11656 raft_consensus.cc:729] T e0583fa43994452ca7553cbf2309eba1 P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a505e7cb133c4392bfda8a4a20874ae9, State: Initialized, Role: FOLLOWER
I20240501 07:59:13.131191 11656 consensus_queue.cc:260] T e0583fa43994452ca7553cbf2309eba1 P a505e7cb133c4392bfda8a4a20874ae9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } }
I20240501 07:59:13.131518 11656 ts_tablet_manager.cc:1398] T e0583fa43994452ca7553cbf2309eba1 P a505e7cb133c4392bfda8a4a20874ae9: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 07:59:13.131691 11656 tablet_bootstrap.cc:492] T 2560b1ec81ac4357b32c7a8ba9d72216 P a505e7cb133c4392bfda8a4a20874ae9: Bootstrap starting.
I20240501 07:59:13.132443 11654 tablet_bootstrap.cc:492] T e0583fa43994452ca7553cbf2309eba1 P 83f20e84af4c40d09a4804b6b26e697f: Bootstrap starting.
I20240501 07:59:13.132550 11656 tablet_bootstrap.cc:654] T 2560b1ec81ac4357b32c7a8ba9d72216 P a505e7cb133c4392bfda8a4a20874ae9: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:13.133920 11654 tablet_bootstrap.cc:654] T e0583fa43994452ca7553cbf2309eba1 P 83f20e84af4c40d09a4804b6b26e697f: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:13.135215 11656 tablet_bootstrap.cc:492] T 2560b1ec81ac4357b32c7a8ba9d72216 P a505e7cb133c4392bfda8a4a20874ae9: No bootstrap required, opened a new log
I20240501 07:59:13.135306 11656 ts_tablet_manager.cc:1367] T 2560b1ec81ac4357b32c7a8ba9d72216 P a505e7cb133c4392bfda8a4a20874ae9: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 07:59:13.135490 11657 tablet_bootstrap.cc:492] T e5bd3a8f7a7640c09774f781009eb19c P 8b8b3ad3aa8f4053b2167dd550f2086f: Bootstrap starting.
I20240501 07:59:13.135691 11658 tablet_bootstrap.cc:492] T 2560b1ec81ac4357b32c7a8ba9d72216 P 74077b61d6f54342bf9736b4a9f2eade: Bootstrap starting.
I20240501 07:59:13.135759 11656 raft_consensus.cc:348] T 2560b1ec81ac4357b32c7a8ba9d72216 P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } }
I20240501 07:59:13.135866 11656 raft_consensus.cc:374] T 2560b1ec81ac4357b32c7a8ba9d72216 P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:13.135902 11656 raft_consensus.cc:729] T 2560b1ec81ac4357b32c7a8ba9d72216 P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a505e7cb133c4392bfda8a4a20874ae9, State: Initialized, Role: FOLLOWER
I20240501 07:59:13.136032 11656 consensus_queue.cc:260] T 2560b1ec81ac4357b32c7a8ba9d72216 P a505e7cb133c4392bfda8a4a20874ae9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } }
I20240501 07:59:13.136298 11656 ts_tablet_manager.cc:1398] T 2560b1ec81ac4357b32c7a8ba9d72216 P a505e7cb133c4392bfda8a4a20874ae9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:13.136431 11657 tablet_bootstrap.cc:654] T e5bd3a8f7a7640c09774f781009eb19c P 8b8b3ad3aa8f4053b2167dd550f2086f: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:13.136461 11656 tablet_bootstrap.cc:492] T e5bd3a8f7a7640c09774f781009eb19c P a505e7cb133c4392bfda8a4a20874ae9: Bootstrap starting.
I20240501 07:59:13.136595 11658 tablet_bootstrap.cc:654] T 2560b1ec81ac4357b32c7a8ba9d72216 P 74077b61d6f54342bf9736b4a9f2eade: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:13.137445 11656 tablet_bootstrap.cc:654] T e5bd3a8f7a7640c09774f781009eb19c P a505e7cb133c4392bfda8a4a20874ae9: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:13.137583 11658 tablet_bootstrap.cc:492] T 2560b1ec81ac4357b32c7a8ba9d72216 P 74077b61d6f54342bf9736b4a9f2eade: No bootstrap required, opened a new log
I20240501 07:59:13.137617 11655 tablet_bootstrap.cc:492] T e0583fa43994452ca7553cbf2309eba1 P 13d05ebdca8c4f489c597ce550dd773d: No bootstrap required, opened a new log
I20240501 07:59:13.137662 11658 ts_tablet_manager.cc:1367] T 2560b1ec81ac4357b32c7a8ba9d72216 P 74077b61d6f54342bf9736b4a9f2eade: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240501 07:59:13.137701 11655 ts_tablet_manager.cc:1367] T e0583fa43994452ca7553cbf2309eba1 P 13d05ebdca8c4f489c597ce550dd773d: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240501 07:59:13.137976 11654 tablet_bootstrap.cc:492] T e0583fa43994452ca7553cbf2309eba1 P 83f20e84af4c40d09a4804b6b26e697f: No bootstrap required, opened a new log
I20240501 07:59:13.138067 11654 ts_tablet_manager.cc:1367] T e0583fa43994452ca7553cbf2309eba1 P 83f20e84af4c40d09a4804b6b26e697f: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 07:59:13.138084 11658 raft_consensus.cc:348] T 2560b1ec81ac4357b32c7a8ba9d72216 P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } }
I20240501 07:59:13.138193 11658 raft_consensus.cc:374] T 2560b1ec81ac4357b32c7a8ba9d72216 P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:13.138134 11655 raft_consensus.cc:348] T e0583fa43994452ca7553cbf2309eba1 P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } }
I20240501 07:59:13.138222 11658 raft_consensus.cc:729] T 2560b1ec81ac4357b32c7a8ba9d72216 P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74077b61d6f54342bf9736b4a9f2eade, State: Initialized, Role: FOLLOWER
I20240501 07:59:13.138231 11655 raft_consensus.cc:374] T e0583fa43994452ca7553cbf2309eba1 P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:13.138264 11655 raft_consensus.cc:729] T e0583fa43994452ca7553cbf2309eba1 P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13d05ebdca8c4f489c597ce550dd773d, State: Initialized, Role: FOLLOWER
I20240501 07:59:13.138355 11658 consensus_queue.cc:260] T 2560b1ec81ac4357b32c7a8ba9d72216 P 74077b61d6f54342bf9736b4a9f2eade [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } }
I20240501 07:59:13.138432 11655 consensus_queue.cc:260] T e0583fa43994452ca7553cbf2309eba1 P 13d05ebdca8c4f489c597ce550dd773d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } }
I20240501 07:59:13.138532 11654 raft_consensus.cc:348] T e0583fa43994452ca7553cbf2309eba1 P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } }
I20240501 07:59:13.138618 11658 ts_tablet_manager.cc:1398] T 2560b1ec81ac4357b32c7a8ba9d72216 P 74077b61d6f54342bf9736b4a9f2eade: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 07:59:13.138649 11654 raft_consensus.cc:374] T e0583fa43994452ca7553cbf2309eba1 P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:13.138684 11654 raft_consensus.cc:729] T e0583fa43994452ca7553cbf2309eba1 P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83f20e84af4c40d09a4804b6b26e697f, State: Initialized, Role: FOLLOWER
I20240501 07:59:13.138684 11655 ts_tablet_manager.cc:1398] T e0583fa43994452ca7553cbf2309eba1 P 13d05ebdca8c4f489c597ce550dd773d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:13.138783 11658 tablet_bootstrap.cc:492] T e5bd3a8f7a7640c09774f781009eb19c P 74077b61d6f54342bf9736b4a9f2eade: Bootstrap starting.
I20240501 07:59:13.138865 11654 consensus_queue.cc:260] T e0583fa43994452ca7553cbf2309eba1 P 83f20e84af4c40d09a4804b6b26e697f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } }
I20240501 07:59:13.139174 11654 ts_tablet_manager.cc:1398] T e0583fa43994452ca7553cbf2309eba1 P 83f20e84af4c40d09a4804b6b26e697f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:13.139592 11658 tablet_bootstrap.cc:654] T e5bd3a8f7a7640c09774f781009eb19c P 74077b61d6f54342bf9736b4a9f2eade: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:13.140599 11658 tablet_bootstrap.cc:492] T e5bd3a8f7a7640c09774f781009eb19c P 74077b61d6f54342bf9736b4a9f2eade: No bootstrap required, opened a new log
I20240501 07:59:13.141584 11657 tablet_bootstrap.cc:492] T e5bd3a8f7a7640c09774f781009eb19c P 8b8b3ad3aa8f4053b2167dd550f2086f: No bootstrap required, opened a new log
I20240501 07:59:13.141682 11657 ts_tablet_manager.cc:1367] T e5bd3a8f7a7640c09774f781009eb19c P 8b8b3ad3aa8f4053b2167dd550f2086f: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 07:59:13.140667 11658 ts_tablet_manager.cc:1367] T e5bd3a8f7a7640c09774f781009eb19c P 74077b61d6f54342bf9736b4a9f2eade: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240501 07:59:13.142346 11658 raft_consensus.cc:348] T e5bd3a8f7a7640c09774f781009eb19c P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } }
I20240501 07:59:13.142455 11658 raft_consensus.cc:374] T e5bd3a8f7a7640c09774f781009eb19c P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:13.142488 11658 raft_consensus.cc:729] T e5bd3a8f7a7640c09774f781009eb19c P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74077b61d6f54342bf9736b4a9f2eade, State: Initialized, Role: FOLLOWER
I20240501 07:59:13.142604 11658 consensus_queue.cc:260] T e5bd3a8f7a7640c09774f781009eb19c P 74077b61d6f54342bf9736b4a9f2eade [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } }
I20240501 07:59:13.142931 11658 ts_tablet_manager.cc:1398] T e5bd3a8f7a7640c09774f781009eb19c P 74077b61d6f54342bf9736b4a9f2eade: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:59:13.143272 11657 raft_consensus.cc:348] T e5bd3a8f7a7640c09774f781009eb19c P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } }
I20240501 07:59:13.143584 11657 raft_consensus.cc:374] T e5bd3a8f7a7640c09774f781009eb19c P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:13.143666 11657 raft_consensus.cc:729] T e5bd3a8f7a7640c09774f781009eb19c P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b8b3ad3aa8f4053b2167dd550f2086f, State: Initialized, Role: FOLLOWER
I20240501 07:59:13.143850 11657 consensus_queue.cc:260] T e5bd3a8f7a7640c09774f781009eb19c P 8b8b3ad3aa8f4053b2167dd550f2086f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } }
I20240501 07:59:13.145671 11657 ts_tablet_manager.cc:1398] T e5bd3a8f7a7640c09774f781009eb19c P 8b8b3ad3aa8f4053b2167dd550f2086f: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 07:59:13.148335 11656 tablet_bootstrap.cc:492] T e5bd3a8f7a7640c09774f781009eb19c P a505e7cb133c4392bfda8a4a20874ae9: No bootstrap required, opened a new log
I20240501 07:59:13.148443 11656 ts_tablet_manager.cc:1367] T e5bd3a8f7a7640c09774f781009eb19c P a505e7cb133c4392bfda8a4a20874ae9: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.002s
I20240501 07:59:13.148890 11656 raft_consensus.cc:348] T e5bd3a8f7a7640c09774f781009eb19c P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } }
I20240501 07:59:13.149049 11656 raft_consensus.cc:374] T e5bd3a8f7a7640c09774f781009eb19c P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:13.149091 11656 raft_consensus.cc:729] T e5bd3a8f7a7640c09774f781009eb19c P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a505e7cb133c4392bfda8a4a20874ae9, State: Initialized, Role: FOLLOWER
I20240501 07:59:13.149217 11656 consensus_queue.cc:260] T e5bd3a8f7a7640c09774f781009eb19c P a505e7cb133c4392bfda8a4a20874ae9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } }
I20240501 07:59:13.149488 11656 ts_tablet_manager.cc:1398] T e5bd3a8f7a7640c09774f781009eb19c P a505e7cb133c4392bfda8a4a20874ae9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:13.169641 11687 ts_tablet_manager.cc:902] T ff9dc384dbac47dc9795617c64688943 P 83f20e84af4c40d09a4804b6b26e697f: Initiating tablet copy from peer 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132:36773)
I20240501 07:59:13.170189 11687 tablet_copy_client.cc:308] T ff9dc384dbac47dc9795617c64688943 P 83f20e84af4c40d09a4804b6b26e697f: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:36773
I20240501 07:59:13.170660 6728 tablet_copy_service.cc:140] P 1f9cb72be98941b5a8e504cbe34132b9: Received BeginTabletCopySession request for tablet ff9dc384dbac47dc9795617c64688943 from peer 83f20e84af4c40d09a4804b6b26e697f ({username='slave'} at 127.0.99.136:55471)
I20240501 07:59:13.170778 6728 tablet_copy_service.cc:161] P 1f9cb72be98941b5a8e504cbe34132b9: Beginning new tablet copy session on tablet ff9dc384dbac47dc9795617c64688943 from peer 83f20e84af4c40d09a4804b6b26e697f at {username='slave'} at 127.0.99.136:55471: session id = 83f20e84af4c40d09a4804b6b26e697f-ff9dc384dbac47dc9795617c64688943
I20240501 07:59:13.171801 6728 tablet_copy_source_session.cc:205] T ff9dc384dbac47dc9795617c64688943 P 1f9cb72be98941b5a8e504cbe34132b9: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:13.173590 11687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff9dc384dbac47dc9795617c64688943. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:13.175861 11687 tablet_copy_client.cc:783] T ff9dc384dbac47dc9795617c64688943 P 83f20e84af4c40d09a4804b6b26e697f: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:13.176105 11687 tablet_copy_client.cc:647] T ff9dc384dbac47dc9795617c64688943 P 83f20e84af4c40d09a4804b6b26e697f: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:13.180051 11687 tablet_copy_client.cc:523] T ff9dc384dbac47dc9795617c64688943 P 83f20e84af4c40d09a4804b6b26e697f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:13.182804 11317 ts_tablet_manager.cc:902] T d6c28cd91f624673b8e428f471a1dcce P a505e7cb133c4392bfda8a4a20874ae9: Initiating tablet copy from peer 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131:34471)
I20240501 07:59:13.183365 11317 tablet_copy_client.cc:308] T d6c28cd91f624673b8e428f471a1dcce P a505e7cb133c4392bfda8a4a20874ae9: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:34471
I20240501 07:59:13.183991 6597 tablet_copy_service.cc:140] P 3b164cbbc00f4e8eb3254ddd349a3ff9: Received BeginTabletCopySession request for tablet d6c28cd91f624673b8e428f471a1dcce from peer a505e7cb133c4392bfda8a4a20874ae9 ({username='slave'} at 127.0.99.135:40821)
I20240501 07:59:13.184101 6597 tablet_copy_service.cc:161] P 3b164cbbc00f4e8eb3254ddd349a3ff9: Beginning new tablet copy session on tablet d6c28cd91f624673b8e428f471a1dcce from peer a505e7cb133c4392bfda8a4a20874ae9 at {username='slave'} at 127.0.99.135:40821: session id = a505e7cb133c4392bfda8a4a20874ae9-d6c28cd91f624673b8e428f471a1dcce
I20240501 07:59:13.185171 6597 tablet_copy_source_session.cc:205] T d6c28cd91f624673b8e428f471a1dcce P 3b164cbbc00f4e8eb3254ddd349a3ff9: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:13.187173 11317 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6c28cd91f624673b8e428f471a1dcce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:13.189585 11317 tablet_copy_client.cc:783] T d6c28cd91f624673b8e428f471a1dcce P a505e7cb133c4392bfda8a4a20874ae9: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:13.190732 11317 tablet_copy_client.cc:647] T d6c28cd91f624673b8e428f471a1dcce P a505e7cb133c4392bfda8a4a20874ae9: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:13.182673 11687 tablet_bootstrap.cc:492] T ff9dc384dbac47dc9795617c64688943 P 83f20e84af4c40d09a4804b6b26e697f: Bootstrap starting.
I20240501 07:59:13.193832 10945 raft_consensus.cc:482] T e5bd3a8f7a7640c09774f781009eb19c P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:13.193933 10945 raft_consensus.cc:504] T e5bd3a8f7a7640c09774f781009eb19c P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } }
I20240501 07:59:13.194499 10945 leader_election.cc:288] T e5bd3a8f7a7640c09774f781009eb19c P a505e7cb133c4392bfda8a4a20874ae9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129:46875), 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133:39859)
I20240501 07:59:13.195888 6167 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55950:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 07:59:13.196045 6313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5bd3a8f7a7640c09774f781009eb19c" candidate_uuid: "a505e7cb133c4392bfda8a4a20874ae9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" is_pre_election: true
I20240501 07:59:13.196069 8793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5bd3a8f7a7640c09774f781009eb19c" candidate_uuid: "a505e7cb133c4392bfda8a4a20874ae9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74077b61d6f54342bf9736b4a9f2eade" is_pre_election: true
I20240501 07:59:13.196187 6313 raft_consensus.cc:2454] T e5bd3a8f7a7640c09774f781009eb19c P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a505e7cb133c4392bfda8a4a20874ae9 in term 0.
I20240501 07:59:13.196213 8793 raft_consensus.cc:2454] T e5bd3a8f7a7640c09774f781009eb19c P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a505e7cb133c4392bfda8a4a20874ae9 in term 0.
I20240501 07:59:13.197356 8989 leader_election.cc:302] T e5bd3a8f7a7640c09774f781009eb19c P a505e7cb133c4392bfda8a4a20874ae9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b8b3ad3aa8f4053b2167dd550f2086f, a505e7cb133c4392bfda8a4a20874ae9; no voters:
I20240501 07:59:13.198043 10945 raft_consensus.cc:2781] T e5bd3a8f7a7640c09774f781009eb19c P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:13.198108 10945 raft_consensus.cc:482] T e5bd3a8f7a7640c09774f781009eb19c P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:13.198148 10945 raft_consensus.cc:3037] T e5bd3a8f7a7640c09774f781009eb19c P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:13.199277 10945 raft_consensus.cc:504] T e5bd3a8f7a7640c09774f781009eb19c P a505e7cb133c4392bfda8a4a20874ae9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } }
I20240501 07:59:13.199638 10945 leader_election.cc:288] T e5bd3a8f7a7640c09774f781009eb19c P a505e7cb133c4392bfda8a4a20874ae9 [CANDIDATE]: Term 1 election: Requested vote from peers 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129:46875), 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133:39859)
I20240501 07:59:13.201184 8793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5bd3a8f7a7640c09774f781009eb19c" candidate_uuid: "a505e7cb133c4392bfda8a4a20874ae9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74077b61d6f54342bf9736b4a9f2eade"
I20240501 07:59:13.201332 8793 raft_consensus.cc:3037] T e5bd3a8f7a7640c09774f781009eb19c P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:13.201545 6313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5bd3a8f7a7640c09774f781009eb19c" candidate_uuid: "a505e7cb133c4392bfda8a4a20874ae9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f"
I20240501 07:59:13.201645 6313 raft_consensus.cc:3037] T e5bd3a8f7a7640c09774f781009eb19c P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:13.202455 8793 raft_consensus.cc:2454] T e5bd3a8f7a7640c09774f781009eb19c P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a505e7cb133c4392bfda8a4a20874ae9 in term 1.
I20240501 07:59:13.202610 6313 raft_consensus.cc:2454] T e5bd3a8f7a7640c09774f781009eb19c P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a505e7cb133c4392bfda8a4a20874ae9 in term 1.
I20240501 07:59:13.203231 8989 leader_election.cc:302] T e5bd3a8f7a7640c09774f781009eb19c P a505e7cb133c4392bfda8a4a20874ae9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b8b3ad3aa8f4053b2167dd550f2086f, a505e7cb133c4392bfda8a4a20874ae9; no voters:
I20240501 07:59:13.203764 10945 raft_consensus.cc:2781] T e5bd3a8f7a7640c09774f781009eb19c P a505e7cb133c4392bfda8a4a20874ae9 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:13.203866 10945 raft_consensus.cc:686] T e5bd3a8f7a7640c09774f781009eb19c P a505e7cb133c4392bfda8a4a20874ae9 [term 1 LEADER]: Becoming Leader. State: Replica: a505e7cb133c4392bfda8a4a20874ae9, State: Running, Role: LEADER
I20240501 07:59:13.204058 10945 consensus_queue.cc:237] T e5bd3a8f7a7640c09774f781009eb19c P a505e7cb133c4392bfda8a4a20874ae9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } }
I20240501 07:59:13.206205 9169 tablet_service.cc:1467] Processing CreateTablet for tablet 3484230eff9c4c25be777d8a0df76258 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=3abb7a980b184a8ea12a505903abf386]), partition=
I20240501 07:59:13.206491 9169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3484230eff9c4c25be777d8a0df76258. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:13.207131 11317 tablet_copy_client.cc:523] T d6c28cd91f624673b8e428f471a1dcce P a505e7cb133c4392bfda8a4a20874ae9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:13.208534 11317 tablet_bootstrap.cc:492] T d6c28cd91f624673b8e428f471a1dcce P a505e7cb133c4392bfda8a4a20874ae9: Bootstrap starting.
I20240501 07:59:13.208701 11654 tablet_bootstrap.cc:492] T 3484230eff9c4c25be777d8a0df76258 P 83f20e84af4c40d09a4804b6b26e697f: Bootstrap starting.
I20240501 07:59:13.209612 11654 tablet_bootstrap.cc:654] T 3484230eff9c4c25be777d8a0df76258 P 83f20e84af4c40d09a4804b6b26e697f: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:13.210196 10478 raft_consensus.cc:482] T e0583fa43994452ca7553cbf2309eba1 P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:13.210297 10478 raft_consensus.cc:504] T e0583fa43994452ca7553cbf2309eba1 P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } }
I20240501 07:59:13.210701 10478 leader_election.cc:288] T e0583fa43994452ca7553cbf2309eba1 P 13d05ebdca8c4f489c597ce550dd773d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149), a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607)
I20240501 07:59:13.211423 9056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0583fa43994452ca7553cbf2309eba1" candidate_uuid: "13d05ebdca8c4f489c597ce550dd773d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a505e7cb133c4392bfda8a4a20874ae9" is_pre_election: true
I20240501 07:59:13.211578 9056 raft_consensus.cc:2454] T e0583fa43994452ca7553cbf2309eba1 P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 13d05ebdca8c4f489c597ce550dd773d in term 0.
I20240501 07:59:13.211892 8859 leader_election.cc:302] T e0583fa43994452ca7553cbf2309eba1 P 13d05ebdca8c4f489c597ce550dd773d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13d05ebdca8c4f489c597ce550dd773d, a505e7cb133c4392bfda8a4a20874ae9; no voters:
I20240501 07:59:13.212145 10478 raft_consensus.cc:2781] T e0583fa43994452ca7553cbf2309eba1 P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:13.212190 10478 raft_consensus.cc:482] T e0583fa43994452ca7553cbf2309eba1 P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:13.212224 10478 raft_consensus.cc:3037] T e0583fa43994452ca7553cbf2309eba1 P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:13.213253 10478 raft_consensus.cc:504] T e0583fa43994452ca7553cbf2309eba1 P 13d05ebdca8c4f489c597ce550dd773d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } }
I20240501 07:59:13.213343 9189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0583fa43994452ca7553cbf2309eba1" candidate_uuid: "13d05ebdca8c4f489c597ce550dd773d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83f20e84af4c40d09a4804b6b26e697f" is_pre_election: true
I20240501 07:59:13.213464 9189 raft_consensus.cc:2454] T e0583fa43994452ca7553cbf2309eba1 P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 13d05ebdca8c4f489c597ce550dd773d in term 0.
I20240501 07:59:13.213660 10478 leader_election.cc:288] T e0583fa43994452ca7553cbf2309eba1 P 13d05ebdca8c4f489c597ce550dd773d [CANDIDATE]: Term 1 election: Requested vote from peers 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149), a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607)
I20240501 07:59:13.214322 9056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0583fa43994452ca7553cbf2309eba1" candidate_uuid: "13d05ebdca8c4f489c597ce550dd773d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a505e7cb133c4392bfda8a4a20874ae9"
I20240501 07:59:13.214399 9188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0583fa43994452ca7553cbf2309eba1" candidate_uuid: "13d05ebdca8c4f489c597ce550dd773d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83f20e84af4c40d09a4804b6b26e697f"
I20240501 07:59:13.214428 9056 raft_consensus.cc:3037] T e0583fa43994452ca7553cbf2309eba1 P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:13.214525 9188 raft_consensus.cc:3037] T e0583fa43994452ca7553cbf2309eba1 P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:13.215400 9056 raft_consensus.cc:2454] T e0583fa43994452ca7553cbf2309eba1 P a505e7cb133c4392bfda8a4a20874ae9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13d05ebdca8c4f489c597ce550dd773d in term 1.
I20240501 07:59:13.215400 9188 raft_consensus.cc:2454] T e0583fa43994452ca7553cbf2309eba1 P 83f20e84af4c40d09a4804b6b26e697f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13d05ebdca8c4f489c597ce550dd773d in term 1.
I20240501 07:59:13.215775 8859 leader_election.cc:302] T e0583fa43994452ca7553cbf2309eba1 P 13d05ebdca8c4f489c597ce550dd773d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13d05ebdca8c4f489c597ce550dd773d, 83f20e84af4c40d09a4804b6b26e697f; no voters:
I20240501 07:59:13.216540 8757 tablet_service.cc:1467] Processing CreateTablet for tablet 3484230eff9c4c25be777d8a0df76258 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=3abb7a980b184a8ea12a505903abf386]), partition=
I20240501 07:59:13.216653 6295 tablet_service.cc:1467] Processing CreateTablet for tablet 3484230eff9c4c25be777d8a0df76258 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=3abb7a980b184a8ea12a505903abf386]), partition=
I20240501 07:59:13.216814 8757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3484230eff9c4c25be777d8a0df76258. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:13.216907 6295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3484230eff9c4c25be777d8a0df76258. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:13.218897 11658 tablet_bootstrap.cc:492] T 3484230eff9c4c25be777d8a0df76258 P 74077b61d6f54342bf9736b4a9f2eade: Bootstrap starting.
I20240501 07:59:13.219379 11657 tablet_bootstrap.cc:492] T 3484230eff9c4c25be777d8a0df76258 P 8b8b3ad3aa8f4053b2167dd550f2086f: Bootstrap starting.
I20240501 07:59:13.219903 11658 tablet_bootstrap.cc:654] T 3484230eff9c4c25be777d8a0df76258 P 74077b61d6f54342bf9736b4a9f2eade: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:13.220150 11657 tablet_bootstrap.cc:654] T 3484230eff9c4c25be777d8a0df76258 P 8b8b3ad3aa8f4053b2167dd550f2086f: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:13.220206 10478 raft_consensus.cc:2781] T e0583fa43994452ca7553cbf2309eba1 P 13d05ebdca8c4f489c597ce550dd773d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:13.220322 10478 raft_consensus.cc:686] T e0583fa43994452ca7553cbf2309eba1 P 13d05ebdca8c4f489c597ce550dd773d [term 1 LEADER]: Becoming Leader. State: Replica: 13d05ebdca8c4f489c597ce550dd773d, State: Running, Role: LEADER
I20240501 07:59:13.220481 10478 consensus_queue.cc:237] T e0583fa43994452ca7553cbf2309eba1 P 13d05ebdca8c4f489c597ce550dd773d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } }
I20240501 07:59:13.222965 6166 catalog_manager.cc:5484] T e5bd3a8f7a7640c09774f781009eb19c P a505e7cb133c4392bfda8a4a20874ae9 reported cstate change: term changed from 0 to 1, leader changed from <none> to a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135). New cstate: current_term: 1 leader_uuid: "a505e7cb133c4392bfda8a4a20874ae9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } health_report { overall_health: UNKNOWN } } }
I20240501 07:59:13.223338 6167 catalog_manager.cc:5484] T e0583fa43994452ca7553cbf2309eba1 P 13d05ebdca8c4f489c597ce550dd773d reported cstate change: term changed from 0 to 1, leader changed from <none> to 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134). New cstate: current_term: 1 leader_uuid: "13d05ebdca8c4f489c597ce550dd773d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } health_report { overall_health: HEALTHY } } }
I20240501 07:59:13.224926 11654 tablet_bootstrap.cc:492] T 3484230eff9c4c25be777d8a0df76258 P 83f20e84af4c40d09a4804b6b26e697f: No bootstrap required, opened a new log
I20240501 07:59:13.225060 11654 ts_tablet_manager.cc:1367] T 3484230eff9c4c25be777d8a0df76258 P 83f20e84af4c40d09a4804b6b26e697f: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.000s
I20240501 07:59:13.225549 11654 raft_consensus.cc:348] T 3484230eff9c4c25be777d8a0df76258 P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } }
I20240501 07:59:13.225652 11654 raft_consensus.cc:374] T 3484230eff9c4c25be777d8a0df76258 P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:13.225682 11654 raft_consensus.cc:729] T 3484230eff9c4c25be777d8a0df76258 P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83f20e84af4c40d09a4804b6b26e697f, State: Initialized, Role: FOLLOWER
I20240501 07:59:13.225800 11654 consensus_queue.cc:260] T 3484230eff9c4c25be777d8a0df76258 P 83f20e84af4c40d09a4804b6b26e697f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } }
I20240501 07:59:13.226109 11654 ts_tablet_manager.cc:1398] T 3484230eff9c4c25be777d8a0df76258 P 83f20e84af4c40d09a4804b6b26e697f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:13.231362 11657 tablet_bootstrap.cc:492] T 3484230eff9c4c25be777d8a0df76258 P 8b8b3ad3aa8f4053b2167dd550f2086f: No bootstrap required, opened a new log
I20240501 07:59:13.231472 11657 ts_tablet_manager.cc:1367] T 3484230eff9c4c25be777d8a0df76258 P 8b8b3ad3aa8f4053b2167dd550f2086f: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240501 07:59:13.232015 11657 raft_consensus.cc:348] T 3484230eff9c4c25be777d8a0df76258 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } }
I20240501 07:59:13.232142 11657 raft_consensus.cc:374] T 3484230eff9c4c25be777d8a0df76258 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:13.232183 11657 raft_consensus.cc:729] T 3484230eff9c4c25be777d8a0df76258 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b8b3ad3aa8f4053b2167dd550f2086f, State: Initialized, Role: FOLLOWER
I20240501 07:59:13.232126 11712 ts_tablet_manager.cc:902] T dc550f54e8024d47b1955905e7858a1e P 74077b61d6f54342bf9736b4a9f2eade: Initiating tablet copy from peer 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131:34471)
I20240501 07:59:13.232339 11657 consensus_queue.cc:260] T 3484230eff9c4c25be777d8a0df76258 P 8b8b3ad3aa8f4053b2167dd550f2086f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } }
I20240501 07:59:13.232620 11657 ts_tablet_manager.cc:1398] T 3484230eff9c4c25be777d8a0df76258 P 8b8b3ad3aa8f4053b2167dd550f2086f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:13.233096 11712 tablet_copy_client.cc:308] T dc550f54e8024d47b1955905e7858a1e P 74077b61d6f54342bf9736b4a9f2eade: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:34471
I20240501 07:59:13.237320 6597 tablet_copy_service.cc:140] P 3b164cbbc00f4e8eb3254ddd349a3ff9: Received BeginTabletCopySession request for tablet dc550f54e8024d47b1955905e7858a1e from peer 74077b61d6f54342bf9736b4a9f2eade ({username='slave'} at 127.0.99.133:57559)
I20240501 07:59:13.237442 6597 tablet_copy_service.cc:161] P 3b164cbbc00f4e8eb3254ddd349a3ff9: Beginning new tablet copy session on tablet dc550f54e8024d47b1955905e7858a1e from peer 74077b61d6f54342bf9736b4a9f2eade at {username='slave'} at 127.0.99.133:57559: session id = 74077b61d6f54342bf9736b4a9f2eade-dc550f54e8024d47b1955905e7858a1e
I20240501 07:59:13.238471 6597 tablet_copy_source_session.cc:205] T dc550f54e8024d47b1955905e7858a1e P 3b164cbbc00f4e8eb3254ddd349a3ff9: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:13.239404 11712 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc550f54e8024d47b1955905e7858a1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:13.242805 11712 tablet_copy_client.cc:783] T dc550f54e8024d47b1955905e7858a1e P 74077b61d6f54342bf9736b4a9f2eade: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:13.243052 11712 tablet_copy_client.cc:647] T dc550f54e8024d47b1955905e7858a1e P 74077b61d6f54342bf9736b4a9f2eade: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:13.244732 11712 tablet_copy_client.cc:523] T dc550f54e8024d47b1955905e7858a1e P 74077b61d6f54342bf9736b4a9f2eade: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:13.245466 11658 tablet_bootstrap.cc:492] T 3484230eff9c4c25be777d8a0df76258 P 74077b61d6f54342bf9736b4a9f2eade: No bootstrap required, opened a new log
I20240501 07:59:13.245579 11658 ts_tablet_manager.cc:1367] T 3484230eff9c4c25be777d8a0df76258 P 74077b61d6f54342bf9736b4a9f2eade: Time spent bootstrapping tablet: real 0.027s user 0.000s sys 0.003s
I20240501 07:59:13.246079 11658 raft_consensus.cc:348] T 3484230eff9c4c25be777d8a0df76258 P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } }
I20240501 07:59:13.246191 11658 raft_consensus.cc:374] T 3484230eff9c4c25be777d8a0df76258 P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:13.246220 11658 raft_consensus.cc:729] T 3484230eff9c4c25be777d8a0df76258 P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74077b61d6f54342bf9736b4a9f2eade, State: Initialized, Role: FOLLOWER
I20240501 07:59:13.246378 11658 consensus_queue.cc:260] T 3484230eff9c4c25be777d8a0df76258 P 74077b61d6f54342bf9736b4a9f2eade [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } }
I20240501 07:59:13.246588 11712 tablet_bootstrap.cc:492] T dc550f54e8024d47b1955905e7858a1e P 74077b61d6f54342bf9736b4a9f2eade: Bootstrap starting.
I20240501 07:59:13.246659 11658 ts_tablet_manager.cc:1398] T 3484230eff9c4c25be777d8a0df76258 P 74077b61d6f54342bf9736b4a9f2eade: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:59:13.255940 10324 raft_consensus.cc:1050] T 8454f24b0c28451b8ee51be39ca3e07f P 1f9cb72be98941b5a8e504cbe34132b9: attempting to promote NON_VOTER 13d05ebdca8c4f489c597ce550dd773d to VOTER
I20240501 07:59:13.256461 10324 consensus_queue.cc:237] T 8454f24b0c28451b8ee51be39ca3e07f P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false } }
I20240501 07:59:13.258266 6313 raft_consensus.cc:1261] T 8454f24b0c28451b8ee51be39ca3e07f P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Refusing update from remote peer 1f9cb72be98941b5a8e504cbe34132b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 07:59:13.259017 11171 consensus_queue.cc:1035] T 8454f24b0c28451b8ee51be39ca3e07f P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 07:59:13.260272 9056 raft_consensus.cc:1261] T 8454f24b0c28451b8ee51be39ca3e07f P a505e7cb133c4392bfda8a4a20874ae9 [term 1 FOLLOWER]: Refusing update from remote peer 1f9cb72be98941b5a8e504cbe34132b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 07:59:13.261564 11171 consensus_queue.cc:1035] T 8454f24b0c28451b8ee51be39ca3e07f P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 07:59:13.262544 11687 tablet_bootstrap.cc:492] T ff9dc384dbac47dc9795617c64688943 P 83f20e84af4c40d09a4804b6b26e697f: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:13.262784 10403 raft_consensus.cc:1050] T 72aa4dc7664b48b7a73d34feba7d44d3 P 74077b61d6f54342bf9736b4a9f2eade: attempting to promote NON_VOTER 8315d22c56f7439a9e1fc82f153f018c to VOTER
I20240501 07:59:13.262919 8925 raft_consensus.cc:1261] T 8454f24b0c28451b8ee51be39ca3e07f P 13d05ebdca8c4f489c597ce550dd773d [term 1 LEARNER]: Refusing update from remote peer 1f9cb72be98941b5a8e504cbe34132b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 07:59:13.263146 10403 consensus_queue.cc:237] T 72aa4dc7664b48b7a73d34feba7d44d3 P 74077b61d6f54342bf9736b4a9f2eade [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { replace: true } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { promote: false } }
I20240501 07:59:13.264375 11171 consensus_queue.cc:1035] T 8454f24b0c28451b8ee51be39ca3e07f P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 07:59:13.264847 6444 raft_consensus.cc:1261] T 72aa4dc7664b48b7a73d34feba7d44d3 P 8315d22c56f7439a9e1fc82f153f018c [term 1 LEARNER]: Refusing update from remote peer 74077b61d6f54342bf9736b4a9f2eade: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 07:59:13.265246 10069 consensus_queue.cc:1035] T 72aa4dc7664b48b7a73d34feba7d44d3 P 74077b61d6f54342bf9736b4a9f2eade [LEADER]: Connected to new peer: Peer: permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 07:59:13.266275 9055 raft_consensus.cc:1261] T 72aa4dc7664b48b7a73d34feba7d44d3 P a505e7cb133c4392bfda8a4a20874ae9 [term 1 FOLLOWER]: Refusing update from remote peer 74077b61d6f54342bf9736b4a9f2eade: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 07:59:13.266692 10069 consensus_queue.cc:1035] T 72aa4dc7664b48b7a73d34feba7d44d3 P 74077b61d6f54342bf9736b4a9f2eade [LEADER]: Connected to new peer: Peer: permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 07:59:13.268694 8924 raft_consensus.cc:1261] T 72aa4dc7664b48b7a73d34feba7d44d3 P 13d05ebdca8c4f489c597ce550dd773d [term 1 FOLLOWER]: Refusing update from remote peer 74077b61d6f54342bf9736b4a9f2eade: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 07:59:13.269136 10069 consensus_queue.cc:1035] T 72aa4dc7664b48b7a73d34feba7d44d3 P 74077b61d6f54342bf9736b4a9f2eade [LEADER]: Connected to new peer: Peer: permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 07:59:13.269654 11687 tablet_bootstrap.cc:492] T ff9dc384dbac47dc9795617c64688943 P 83f20e84af4c40d09a4804b6b26e697f: Bootstrap complete.
I20240501 07:59:13.269898 11687 ts_tablet_manager.cc:1367] T ff9dc384dbac47dc9795617c64688943 P 83f20e84af4c40d09a4804b6b26e697f: Time spent bootstrapping tablet: real 0.088s user 0.026s sys 0.004s
I20240501 07:59:13.270424 11687 raft_consensus.cc:348] T ff9dc384dbac47dc9795617c64688943 P 83f20e84af4c40d09a4804b6b26e697f [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { replace: true } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: true } }
I20240501 07:59:13.270650 11687 raft_consensus.cc:729] T ff9dc384dbac47dc9795617c64688943 P 83f20e84af4c40d09a4804b6b26e697f [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 83f20e84af4c40d09a4804b6b26e697f, State: Initialized, Role: LEARNER
I20240501 07:59:13.270828 11687 consensus_queue.cc:260] T ff9dc384dbac47dc9795617c64688943 P 83f20e84af4c40d09a4804b6b26e697f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { replace: true } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: true } }
I20240501 07:59:13.271198 11687 ts_tablet_manager.cc:1398] T ff9dc384dbac47dc9795617c64688943 P 83f20e84af4c40d09a4804b6b26e697f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:13.272022 6728 tablet_copy_service.cc:342] P 1f9cb72be98941b5a8e504cbe34132b9: Request end of tablet copy session 83f20e84af4c40d09a4804b6b26e697f-ff9dc384dbac47dc9795617c64688943 received from {username='slave'} at 127.0.99.136:55471
I20240501 07:59:13.272111 6728 tablet_copy_service.cc:433] P 1f9cb72be98941b5a8e504cbe34132b9: ending tablet copy session 83f20e84af4c40d09a4804b6b26e697f-ff9dc384dbac47dc9795617c64688943 on tablet ff9dc384dbac47dc9795617c64688943 with peer 83f20e84af4c40d09a4804b6b26e697f
I20240501 07:59:13.273855 11726 ts_tablet_manager.cc:902] T f88fe033bce14bf0b1b63a52a7ba137e P 8315d22c56f7439a9e1fc82f153f018c: Initiating tablet copy from peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607)
I20240501 07:59:13.274348 11726 tablet_copy_client.cc:308] T f88fe033bce14bf0b1b63a52a7ba137e P 8315d22c56f7439a9e1fc82f153f018c: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.135:42607
I20240501 07:59:13.274855 9055 raft_consensus.cc:2932] T 72aa4dc7664b48b7a73d34feba7d44d3 P a505e7cb133c4392bfda8a4a20874ae9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { replace: true } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { promote: false } } }
I20240501 07:59:13.274966 9076 tablet_copy_service.cc:140] P a505e7cb133c4392bfda8a4a20874ae9: Received BeginTabletCopySession request for tablet f88fe033bce14bf0b1b63a52a7ba137e from peer 8315d22c56f7439a9e1fc82f153f018c ({username='slave'} at 127.0.99.130:45059)
I20240501 07:59:13.277267 9076 tablet_copy_service.cc:161] P a505e7cb133c4392bfda8a4a20874ae9: Beginning new tablet copy session on tablet f88fe033bce14bf0b1b63a52a7ba137e from peer 8315d22c56f7439a9e1fc82f153f018c at {username='slave'} at 127.0.99.130:45059: session id = 8315d22c56f7439a9e1fc82f153f018c-f88fe033bce14bf0b1b63a52a7ba137e
I20240501 07:59:13.278615 9076 tablet_copy_source_session.cc:205] T f88fe033bce14bf0b1b63a52a7ba137e P a505e7cb133c4392bfda8a4a20874ae9: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:13.279664 10069 raft_consensus.cc:2932] T 72aa4dc7664b48b7a73d34feba7d44d3 P 74077b61d6f54342bf9736b4a9f2eade [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { replace: true } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { promote: false } } }
I20240501 07:59:13.280021 11726 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f88fe033bce14bf0b1b63a52a7ba137e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:13.280210 6167 catalog_manager.cc:5484] T 72aa4dc7664b48b7a73d34feba7d44d3 P a505e7cb133c4392bfda8a4a20874ae9 reported cstate change: config changed from index 2 to 3, 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "74077b61d6f54342bf9736b4a9f2eade" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { replace: true } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { promote: false } } }
I20240501 07:59:13.281404 10945 raft_consensus.cc:482] T 2560b1ec81ac4357b32c7a8ba9d72216 P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:13.281491 10945 raft_consensus.cc:504] T 2560b1ec81ac4357b32c7a8ba9d72216 P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } }
I20240501 07:59:13.281616 6444 raft_consensus.cc:2932] T 72aa4dc7664b48b7a73d34feba7d44d3 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { replace: true } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { promote: false } } }
I20240501 07:59:13.281863 10945 leader_election.cc:288] T 2560b1ec81ac4357b32c7a8ba9d72216 P a505e7cb133c4392bfda8a4a20874ae9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132:36773), 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133:39859)
I20240501 07:59:13.282163 6707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2560b1ec81ac4357b32c7a8ba9d72216" candidate_uuid: "a505e7cb133c4392bfda8a4a20874ae9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f9cb72be98941b5a8e504cbe34132b9" is_pre_election: true
I20240501 07:59:13.282338 6707 raft_consensus.cc:2454] T 2560b1ec81ac4357b32c7a8ba9d72216 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a505e7cb133c4392bfda8a4a20874ae9 in term 0.
I20240501 07:59:13.283938 11171 raft_consensus.cc:2932] T 8454f24b0c28451b8ee51be39ca3e07f P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false } } }
I20240501 07:59:13.284404 8925 raft_consensus.cc:1203] T df00e37199e944589a0a5f3feeee308a P 13d05ebdca8c4f489c597ce550dd773d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 07:59:13.285604 8991 leader_election.cc:302] T 2560b1ec81ac4357b32c7a8ba9d72216 P a505e7cb133c4392bfda8a4a20874ae9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f9cb72be98941b5a8e504cbe34132b9, a505e7cb133c4392bfda8a4a20874ae9; no voters:
I20240501 07:59:13.285810 10945 raft_consensus.cc:2781] T 2560b1ec81ac4357b32c7a8ba9d72216 P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:13.285858 10945 raft_consensus.cc:482] T 2560b1ec81ac4357b32c7a8ba9d72216 P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:13.285892 10945 raft_consensus.cc:3037] T 2560b1ec81ac4357b32c7a8ba9d72216 P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:13.286839 10945 raft_consensus.cc:504] T 2560b1ec81ac4357b32c7a8ba9d72216 P a505e7cb133c4392bfda8a4a20874ae9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } }
I20240501 07:59:13.287245 10945 leader_election.cc:288] T 2560b1ec81ac4357b32c7a8ba9d72216 P a505e7cb133c4392bfda8a4a20874ae9 [CANDIDATE]: Term 1 election: Requested vote from peers 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132:36773), 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133:39859)
I20240501 07:59:13.287307 6165 catalog_manager.cc:5484] T 8454f24b0c28451b8ee51be39ca3e07f P 1f9cb72be98941b5a8e504cbe34132b9 reported cstate change: config changed from index 7 to 8, 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1f9cb72be98941b5a8e504cbe34132b9" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 07:59:13.287415 9055 raft_consensus.cc:2932] T 8454f24b0c28451b8ee51be39ca3e07f P a505e7cb133c4392bfda8a4a20874ae9 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false } } }
I20240501 07:59:13.289374 8794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2560b1ec81ac4357b32c7a8ba9d72216" candidate_uuid: "a505e7cb133c4392bfda8a4a20874ae9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74077b61d6f54342bf9736b4a9f2eade" is_pre_election: true
I20240501 07:59:13.289469 6707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2560b1ec81ac4357b32c7a8ba9d72216" candidate_uuid: "a505e7cb133c4392bfda8a4a20874ae9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f9cb72be98941b5a8e504cbe34132b9"
I20240501 07:59:13.289387 8792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2560b1ec81ac4357b32c7a8ba9d72216" candidate_uuid: "a505e7cb133c4392bfda8a4a20874ae9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74077b61d6f54342bf9736b4a9f2eade"
I20240501 07:59:13.289587 6707 raft_consensus.cc:3037] T 2560b1ec81ac4357b32c7a8ba9d72216 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:13.289511 8794 raft_consensus.cc:2454] T 2560b1ec81ac4357b32c7a8ba9d72216 P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a505e7cb133c4392bfda8a4a20874ae9 in term 0.
I20240501 07:59:13.290076 11726 tablet_copy_client.cc:783] T f88fe033bce14bf0b1b63a52a7ba137e P 8315d22c56f7439a9e1fc82f153f018c: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:13.290372 11726 tablet_copy_client.cc:647] T f88fe033bce14bf0b1b63a52a7ba137e P 8315d22c56f7439a9e1fc82f153f018c: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:13.290597 6707 raft_consensus.cc:2454] T 2560b1ec81ac4357b32c7a8ba9d72216 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a505e7cb133c4392bfda8a4a20874ae9 in term 1.
I20240501 07:59:13.291083 8991 leader_election.cc:302] T 2560b1ec81ac4357b32c7a8ba9d72216 P a505e7cb133c4392bfda8a4a20874ae9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f9cb72be98941b5a8e504cbe34132b9, a505e7cb133c4392bfda8a4a20874ae9; no voters:
I20240501 07:59:13.291477 10945 raft_consensus.cc:2781] T 2560b1ec81ac4357b32c7a8ba9d72216 P a505e7cb133c4392bfda8a4a20874ae9 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:13.291565 10945 raft_consensus.cc:686] T 2560b1ec81ac4357b32c7a8ba9d72216 P a505e7cb133c4392bfda8a4a20874ae9 [term 1 LEADER]: Becoming Leader. State: Replica: a505e7cb133c4392bfda8a4a20874ae9, State: Running, Role: LEADER
I20240501 07:59:13.291687 10945 consensus_queue.cc:237] T 2560b1ec81ac4357b32c7a8ba9d72216 P a505e7cb133c4392bfda8a4a20874ae9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } }
I20240501 07:59:13.295130 8925 raft_consensus.cc:2932] T 8454f24b0c28451b8ee51be39ca3e07f P 13d05ebdca8c4f489c597ce550dd773d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false } } }
I20240501 07:59:13.296936 6707 consensus_queue.cc:237] T 8454f24b0c28451b8ee51be39ca3e07f P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false } }
I20240501 07:59:13.297179 11726 tablet_copy_client.cc:523] T f88fe033bce14bf0b1b63a52a7ba137e P 8315d22c56f7439a9e1fc82f153f018c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:13.298310 8925 raft_consensus.cc:1261] T 8454f24b0c28451b8ee51be39ca3e07f P 13d05ebdca8c4f489c597ce550dd773d [term 1 FOLLOWER]: Refusing update from remote peer 1f9cb72be98941b5a8e504cbe34132b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 07:59:13.298511 9055 raft_consensus.cc:1261] T 8454f24b0c28451b8ee51be39ca3e07f P a505e7cb133c4392bfda8a4a20874ae9 [term 1 FOLLOWER]: Refusing update from remote peer 1f9cb72be98941b5a8e504cbe34132b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 07:59:13.298969 11726 tablet_bootstrap.cc:492] T f88fe033bce14bf0b1b63a52a7ba137e P 8315d22c56f7439a9e1fc82f153f018c: Bootstrap starting.
I20240501 07:59:13.299167 10324 consensus_queue.cc:1035] T 8454f24b0c28451b8ee51be39ca3e07f P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 07:59:13.299424 11402 consensus_queue.cc:1035] T 8454f24b0c28451b8ee51be39ca3e07f P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 07:59:13.300992 11171 raft_consensus.cc:2932] T 8454f24b0c28451b8ee51be39ca3e07f P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false } } }
I20240501 07:59:13.301755 9055 raft_consensus.cc:2932] T 8454f24b0c28451b8ee51be39ca3e07f P a505e7cb133c4392bfda8a4a20874ae9 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false } } }
I20240501 07:59:13.301787 6167 catalog_manager.cc:5484] T 2560b1ec81ac4357b32c7a8ba9d72216 P a505e7cb133c4392bfda8a4a20874ae9 reported cstate change: term changed from 0 to 1, leader changed from <none> to a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135). New cstate: current_term: 1 leader_uuid: "a505e7cb133c4392bfda8a4a20874ae9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } health_report { overall_health: HEALTHY } } }
I20240501 07:59:13.302712 6159 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8454f24b0c28451b8ee51be39ca3e07f with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 07:59:13.303709 8925 raft_consensus.cc:2932] T 8454f24b0c28451b8ee51be39ca3e07f P 13d05ebdca8c4f489c597ce550dd773d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false } } }
I20240501 07:59:13.304589 6165 catalog_manager.cc:5484] T 8454f24b0c28451b8ee51be39ca3e07f P 1f9cb72be98941b5a8e504cbe34132b9 reported cstate change: config changed from index 8 to 9, VOTER 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "1f9cb72be98941b5a8e504cbe34132b9" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 07:59:13.307009 6295 tablet_service.cc:1514] Processing DeleteTablet for tablet 8454f24b0c28451b8ee51be39ca3e07f with delete_type TABLET_DATA_TOMBSTONED (TS 8b8b3ad3aa8f4053b2167dd550f2086f not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:45480
I20240501 07:59:13.307286 10309 tablet_replica.cc:330] T 8454f24b0c28451b8ee51be39ca3e07f P 8b8b3ad3aa8f4053b2167dd550f2086f: stopping tablet replica
I20240501 07:59:13.307451 10309 raft_consensus.cc:2229] T 8454f24b0c28451b8ee51be39ca3e07f P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:13.308743 10309 pending_rounds.cc:62] T 8454f24b0c28451b8ee51be39ca3e07f P 8b8b3ad3aa8f4053b2167dd550f2086f: Trying to abort 1 pending ops.
I20240501 07:59:13.308799 10309 pending_rounds.cc:68] T 8454f24b0c28451b8ee51be39ca3e07f P 8b8b3ad3aa8f4053b2167dd550f2086f: Aborting op as it isn't in flight: id { term: 1 index: 8 } timestamp: 7022798246936899584 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "8454f24b0c28451b8ee51be39ca3e07f" old_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: true } } } new_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false } } } }
I20240501 07:59:13.308943 10309 raft_consensus.cc:2894] T 8454f24b0c28451b8ee51be39ca3e07f P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Aborting config change with OpId 1.8: Aborted: Op aborted
I20240501 07:59:13.309055 10309 raft_consensus.cc:2866] T 8454f24b0c28451b8ee51be39ca3e07f P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240501 07:59:13.309121 10309 raft_consensus.cc:2258] T 8454f24b0c28451b8ee51be39ca3e07f P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:13.309477 11712 tablet_bootstrap.cc:492] T dc550f54e8024d47b1955905e7858a1e P 74077b61d6f54342bf9736b4a9f2eade: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:13.309764 11712 tablet_bootstrap.cc:492] T dc550f54e8024d47b1955905e7858a1e P 74077b61d6f54342bf9736b4a9f2eade: Bootstrap complete.
I20240501 07:59:13.309962 11712 ts_tablet_manager.cc:1367] T dc550f54e8024d47b1955905e7858a1e P 74077b61d6f54342bf9736b4a9f2eade: Time spent bootstrapping tablet: real 0.063s user 0.025s sys 0.001s
I20240501 07:59:13.310495 10309 ts_tablet_manager.cc:1875] T 8454f24b0c28451b8ee51be39ca3e07f P 8b8b3ad3aa8f4053b2167dd550f2086f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:59:13.310545 11712 raft_consensus.cc:348] T dc550f54e8024d47b1955905e7858a1e P 74077b61d6f54342bf9736b4a9f2eade [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: true } }
I20240501 07:59:13.310652 11712 raft_consensus.cc:729] T dc550f54e8024d47b1955905e7858a1e P 74077b61d6f54342bf9736b4a9f2eade [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 74077b61d6f54342bf9736b4a9f2eade, State: Initialized, Role: LEARNER
I20240501 07:59:13.310743 11712 consensus_queue.cc:260] T dc550f54e8024d47b1955905e7858a1e P 74077b61d6f54342bf9736b4a9f2eade [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: true } }
I20240501 07:59:13.311028 11712 ts_tablet_manager.cc:1398] T dc550f54e8024d47b1955905e7858a1e P 74077b61d6f54342bf9736b4a9f2eade: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:13.312745 11726 tablet_bootstrap.cc:492] T f88fe033bce14bf0b1b63a52a7ba137e P 8315d22c56f7439a9e1fc82f153f018c: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:13.312884 10309 ts_tablet_manager.cc:1888] T 8454f24b0c28451b8ee51be39ca3e07f P 8b8b3ad3aa8f4053b2167dd550f2086f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240501 07:59:13.313036 11726 tablet_bootstrap.cc:492] T f88fe033bce14bf0b1b63a52a7ba137e P 8315d22c56f7439a9e1fc82f153f018c: Bootstrap complete.
I20240501 07:59:13.313056 6597 tablet_copy_service.cc:342] P 3b164cbbc00f4e8eb3254ddd349a3ff9: Request end of tablet copy session 74077b61d6f54342bf9736b4a9f2eade-dc550f54e8024d47b1955905e7858a1e received from {username='slave'} at 127.0.99.133:57559
I20240501 07:59:13.313130 6597 tablet_copy_service.cc:433] P 3b164cbbc00f4e8eb3254ddd349a3ff9: ending tablet copy session 74077b61d6f54342bf9736b4a9f2eade-dc550f54e8024d47b1955905e7858a1e on tablet dc550f54e8024d47b1955905e7858a1e with peer 74077b61d6f54342bf9736b4a9f2eade
I20240501 07:59:13.313225 11726 ts_tablet_manager.cc:1367] T f88fe033bce14bf0b1b63a52a7ba137e P 8315d22c56f7439a9e1fc82f153f018c: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.002s
I20240501 07:59:13.312952 10309 log.cc:1196] T 8454f24b0c28451b8ee51be39ca3e07f P 8b8b3ad3aa8f4053b2167dd550f2086f: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/8454f24b0c28451b8ee51be39ca3e07f
I20240501 07:59:13.313722 11726 raft_consensus.cc:348] T f88fe033bce14bf0b1b63a52a7ba137e P 8315d22c56f7439a9e1fc82f153f018c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { promote: true } }
I20240501 07:59:13.313848 11726 raft_consensus.cc:729] T f88fe033bce14bf0b1b63a52a7ba137e P 8315d22c56f7439a9e1fc82f153f018c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8315d22c56f7439a9e1fc82f153f018c, State: Initialized, Role: LEARNER
I20240501 07:59:13.313961 11726 consensus_queue.cc:260] T f88fe033bce14bf0b1b63a52a7ba137e P 8315d22c56f7439a9e1fc82f153f018c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { promote: true } }
I20240501 07:59:13.314193 6157 catalog_manager.cc:4830] TS 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129:46875): tablet 8454f24b0c28451b8ee51be39ca3e07f (table rebalance_test_table_7 [id=8e242b97203b46d0840cb3b6a649e5a7]) successfully deleted
I20240501 07:59:13.314404 11726 ts_tablet_manager.cc:1398] T f88fe033bce14bf0b1b63a52a7ba137e P 8315d22c56f7439a9e1fc82f153f018c: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 07:59:13.315183 9076 tablet_copy_service.cc:342] P a505e7cb133c4392bfda8a4a20874ae9: Request end of tablet copy session 8315d22c56f7439a9e1fc82f153f018c-f88fe033bce14bf0b1b63a52a7ba137e received from {username='slave'} at 127.0.99.130:45059
I20240501 07:59:13.315286 9076 tablet_copy_service.cc:433] P a505e7cb133c4392bfda8a4a20874ae9: ending tablet copy session 8315d22c56f7439a9e1fc82f153f018c-f88fe033bce14bf0b1b63a52a7ba137e on tablet f88fe033bce14bf0b1b63a52a7ba137e with peer 8315d22c56f7439a9e1fc82f153f018c
I20240501 07:59:13.315959 8924 raft_consensus.cc:2932] T 72aa4dc7664b48b7a73d34feba7d44d3 P 13d05ebdca8c4f489c597ce550dd773d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { replace: true } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { promote: false } } }
I20240501 07:59:13.332527 11317 tablet_bootstrap.cc:492] T d6c28cd91f624673b8e428f471a1dcce P a505e7cb133c4392bfda8a4a20874ae9: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:13.332867 11317 tablet_bootstrap.cc:492] T d6c28cd91f624673b8e428f471a1dcce P a505e7cb133c4392bfda8a4a20874ae9: Bootstrap complete.
I20240501 07:59:13.334386 11317 ts_tablet_manager.cc:1367] T d6c28cd91f624673b8e428f471a1dcce P a505e7cb133c4392bfda8a4a20874ae9: Time spent bootstrapping tablet: real 0.126s user 0.028s sys 0.001s
I20240501 07:59:13.334904 11317 raft_consensus.cc:348] T d6c28cd91f624673b8e428f471a1dcce P a505e7cb133c4392bfda8a4a20874ae9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { replace: true } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: true } }
I20240501 07:59:13.335050 11317 raft_consensus.cc:729] T d6c28cd91f624673b8e428f471a1dcce P a505e7cb133c4392bfda8a4a20874ae9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a505e7cb133c4392bfda8a4a20874ae9, State: Initialized, Role: LEARNER
I20240501 07:59:13.335163 11317 consensus_queue.cc:260] T d6c28cd91f624673b8e428f471a1dcce P a505e7cb133c4392bfda8a4a20874ae9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { replace: true } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: true } }
I20240501 07:59:13.336086 11317 ts_tablet_manager.cc:1398] T d6c28cd91f624673b8e428f471a1dcce P a505e7cb133c4392bfda8a4a20874ae9: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:59:13.336895 8905 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 07:59:13.337177 6597 tablet_copy_service.cc:342] P 3b164cbbc00f4e8eb3254ddd349a3ff9: Request end of tablet copy session a505e7cb133c4392bfda8a4a20874ae9-d6c28cd91f624673b8e428f471a1dcce received from {username='slave'} at 127.0.99.135:40821
I20240501 07:59:13.337306 6597 tablet_copy_service.cc:433] P 3b164cbbc00f4e8eb3254ddd349a3ff9: ending tablet copy session a505e7cb133c4392bfda8a4a20874ae9-d6c28cd91f624673b8e428f471a1dcce on tablet d6c28cd91f624673b8e428f471a1dcce with peer a505e7cb133c4392bfda8a4a20874ae9
I20240501 07:59:13.341521 6556 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 07:59:13.342615 6424 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 07:59:13.347471 6295 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 07:59:13.350803 9169 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 07:59:13.351069 8757 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 07:59:13.357148 6687 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 07:59:13.377121 9035 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 07:59:13.377677 6707 raft_consensus.cc:1203] T 239ee596e62d47169eeb6670dc62d59c P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 07:59:13.390498 11630 raft_consensus.cc:482] T 3484230eff9c4c25be777d8a0df76258 P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:13.390625 11630 raft_consensus.cc:504] T 3484230eff9c4c25be777d8a0df76258 P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } }
I20240501 07:59:13.391014 11630 leader_election.cc:288] T 3484230eff9c4c25be777d8a0df76258 P 83f20e84af4c40d09a4804b6b26e697f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133:39859), 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129:46875)
I20240501 07:59:13.391351 8793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3484230eff9c4c25be777d8a0df76258" candidate_uuid: "83f20e84af4c40d09a4804b6b26e697f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74077b61d6f54342bf9736b4a9f2eade" is_pre_election: true
I20240501 07:59:13.391391 6313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3484230eff9c4c25be777d8a0df76258" candidate_uuid: "83f20e84af4c40d09a4804b6b26e697f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" is_pre_election: true
I20240501 07:59:13.391490 8793 raft_consensus.cc:2454] T 3484230eff9c4c25be777d8a0df76258 P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83f20e84af4c40d09a4804b6b26e697f in term 0.
I20240501 07:59:13.391503 6313 raft_consensus.cc:2454] T 3484230eff9c4c25be777d8a0df76258 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83f20e84af4c40d09a4804b6b26e697f in term 0.
I20240501 07:59:13.391786 9122 leader_election.cc:302] T 3484230eff9c4c25be777d8a0df76258 P 83f20e84af4c40d09a4804b6b26e697f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83f20e84af4c40d09a4804b6b26e697f, 8b8b3ad3aa8f4053b2167dd550f2086f; no voters:
I20240501 07:59:13.391959 11630 raft_consensus.cc:2781] T 3484230eff9c4c25be777d8a0df76258 P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:13.392024 11630 raft_consensus.cc:482] T 3484230eff9c4c25be777d8a0df76258 P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:13.392066 11630 raft_consensus.cc:3037] T 3484230eff9c4c25be777d8a0df76258 P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:13.393102 11630 raft_consensus.cc:504] T 3484230eff9c4c25be777d8a0df76258 P 83f20e84af4c40d09a4804b6b26e697f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } }
I20240501 07:59:13.393404 11630 leader_election.cc:288] T 3484230eff9c4c25be777d8a0df76258 P 83f20e84af4c40d09a4804b6b26e697f [CANDIDATE]: Term 1 election: Requested vote from peers 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133:39859), 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129:46875)
I20240501 07:59:13.393764 8793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3484230eff9c4c25be777d8a0df76258" candidate_uuid: "83f20e84af4c40d09a4804b6b26e697f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74077b61d6f54342bf9736b4a9f2eade"
I20240501 07:59:13.393755 6313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3484230eff9c4c25be777d8a0df76258" candidate_uuid: "83f20e84af4c40d09a4804b6b26e697f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f"
I20240501 07:59:13.393870 6313 raft_consensus.cc:3037] T 3484230eff9c4c25be777d8a0df76258 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:13.393870 8793 raft_consensus.cc:3037] T 3484230eff9c4c25be777d8a0df76258 P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:13.394830 8793 raft_consensus.cc:2454] T 3484230eff9c4c25be777d8a0df76258 P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83f20e84af4c40d09a4804b6b26e697f in term 1.
I20240501 07:59:13.394829 6313 raft_consensus.cc:2454] T 3484230eff9c4c25be777d8a0df76258 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83f20e84af4c40d09a4804b6b26e697f in term 1.
I20240501 07:59:13.395160 9122 leader_election.cc:302] T 3484230eff9c4c25be777d8a0df76258 P 83f20e84af4c40d09a4804b6b26e697f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83f20e84af4c40d09a4804b6b26e697f, 8b8b3ad3aa8f4053b2167dd550f2086f; no voters:
I20240501 07:59:13.395313 11630 raft_consensus.cc:2781] T 3484230eff9c4c25be777d8a0df76258 P 83f20e84af4c40d09a4804b6b26e697f [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:13.395387 11630 raft_consensus.cc:686] T 3484230eff9c4c25be777d8a0df76258 P 83f20e84af4c40d09a4804b6b26e697f [term 1 LEADER]: Becoming Leader. State: Replica: 83f20e84af4c40d09a4804b6b26e697f, State: Running, Role: LEADER
I20240501 07:59:13.395499 11630 consensus_queue.cc:237] T 3484230eff9c4c25be777d8a0df76258 P 83f20e84af4c40d09a4804b6b26e697f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } }
I20240501 07:59:13.398116 6167 catalog_manager.cc:5484] T 3484230eff9c4c25be777d8a0df76258 P 83f20e84af4c40d09a4804b6b26e697f reported cstate change: term changed from 0 to 1, leader changed from <none> to 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136). New cstate: current_term: 1 leader_uuid: "83f20e84af4c40d09a4804b6b26e697f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } health_report { overall_health: UNKNOWN } } }
I20240501 07:59:13.404876 11402 raft_consensus.cc:1050] T 7dd6566ccb6e43388146a6dfd1c4a88c P 1f9cb72be98941b5a8e504cbe34132b9: attempting to promote NON_VOTER 13d05ebdca8c4f489c597ce550dd773d to VOTER
I20240501 07:59:13.405344 11402 consensus_queue.cc:237] T 7dd6566ccb6e43388146a6dfd1c4a88c P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false } }
I20240501 07:59:13.406688 6577 raft_consensus.cc:1261] T 7dd6566ccb6e43388146a6dfd1c4a88c P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Refusing update from remote peer 1f9cb72be98941b5a8e504cbe34132b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:59:13.406733 6444 raft_consensus.cc:1261] T 7dd6566ccb6e43388146a6dfd1c4a88c P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Refusing update from remote peer 1f9cb72be98941b5a8e504cbe34132b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:59:13.407054 8924 raft_consensus.cc:1261] T 7dd6566ccb6e43388146a6dfd1c4a88c P 13d05ebdca8c4f489c597ce550dd773d [term 1 LEARNER]: Refusing update from remote peer 1f9cb72be98941b5a8e504cbe34132b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:59:13.407088 11171 consensus_queue.cc:1035] T 7dd6566ccb6e43388146a6dfd1c4a88c P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:59:13.407423 11171 consensus_queue.cc:1035] T 7dd6566ccb6e43388146a6dfd1c4a88c P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:59:13.407539 11402 consensus_queue.cc:1035] T 7dd6566ccb6e43388146a6dfd1c4a88c P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:59:13.413254 11402 raft_consensus.cc:2932] T 7dd6566ccb6e43388146a6dfd1c4a88c P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false } } }
I20240501 07:59:13.414636 8924 raft_consensus.cc:1203] T 8663a511521644009fc360ad92c093a1 P 13d05ebdca8c4f489c597ce550dd773d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 07:59:13.415050 8925 raft_consensus.cc:2932] T 7dd6566ccb6e43388146a6dfd1c4a88c P 13d05ebdca8c4f489c597ce550dd773d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false } } }
I20240501 07:59:13.417155 6577 raft_consensus.cc:2932] T 7dd6566ccb6e43388146a6dfd1c4a88c P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false } } }
I20240501 07:59:13.419489 6167 catalog_manager.cc:5484] T 7dd6566ccb6e43388146a6dfd1c4a88c P 1f9cb72be98941b5a8e504cbe34132b9 reported cstate change: config changed from index 4 to 5, 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1f9cb72be98941b5a8e504cbe34132b9" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 07:59:13.419739 6444 raft_consensus.cc:2932] T 7dd6566ccb6e43388146a6dfd1c4a88c P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false } } }
I20240501 07:59:13.423115 11119 raft_consensus.cc:1050] T d3f2250a62c6429e886753aaffcbe883 P 3b164cbbc00f4e8eb3254ddd349a3ff9: attempting to promote NON_VOTER a505e7cb133c4392bfda8a4a20874ae9 to VOTER
I20240501 07:59:13.423153 6707 consensus_queue.cc:237] T 7dd6566ccb6e43388146a6dfd1c4a88c P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false } }
I20240501 07:59:13.423540 11119 consensus_queue.cc:237] T d3f2250a62c6429e886753aaffcbe883 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { replace: true } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false } }
I20240501 07:59:13.424453 8924 raft_consensus.cc:1261] T 7dd6566ccb6e43388146a6dfd1c4a88c P 13d05ebdca8c4f489c597ce550dd773d [term 1 FOLLOWER]: Refusing update from remote peer 1f9cb72be98941b5a8e504cbe34132b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 07:59:13.425503 9055 raft_consensus.cc:1261] T d3f2250a62c6429e886753aaffcbe883 P a505e7cb133c4392bfda8a4a20874ae9 [term 1 LEARNER]: Refusing update from remote peer 3b164cbbc00f4e8eb3254ddd349a3ff9: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 07:59:13.425513 6444 raft_consensus.cc:1261] T 7dd6566ccb6e43388146a6dfd1c4a88c P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Refusing update from remote peer 1f9cb72be98941b5a8e504cbe34132b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 07:59:13.425513 9189 raft_consensus.cc:1261] T d3f2250a62c6429e886753aaffcbe883 P 83f20e84af4c40d09a4804b6b26e697f [term 1 FOLLOWER]: Refusing update from remote peer 3b164cbbc00f4e8eb3254ddd349a3ff9: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 07:59:13.425935 9507 consensus_queue.cc:1035] T d3f2250a62c6429e886753aaffcbe883 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 07:59:13.426375 10324 consensus_queue.cc:1035] T 7dd6566ccb6e43388146a6dfd1c4a88c P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 07:59:13.426658 11171 consensus_queue.cc:1035] T 7dd6566ccb6e43388146a6dfd1c4a88c P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 07:59:13.427426 6707 raft_consensus.cc:1261] T d3f2250a62c6429e886753aaffcbe883 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Refusing update from remote peer 3b164cbbc00f4e8eb3254ddd349a3ff9: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 07:59:13.427824 10046 consensus_queue.cc:1035] T d3f2250a62c6429e886753aaffcbe883 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 07:59:13.428265 10046 consensus_queue.cc:1035] T d3f2250a62c6429e886753aaffcbe883 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 07:59:13.429972 11107 raft_consensus.cc:2932] T d3f2250a62c6429e886753aaffcbe883 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { replace: true } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false } } }
I20240501 07:59:13.430769 9189 raft_consensus.cc:2932] T d3f2250a62c6429e886753aaffcbe883 P 83f20e84af4c40d09a4804b6b26e697f [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { replace: true } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false } } }
I20240501 07:59:13.432045 9055 raft_consensus.cc:2932] T d3f2250a62c6429e886753aaffcbe883 P a505e7cb133c4392bfda8a4a20874ae9 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { replace: true } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false } } }
I20240501 07:59:13.432775 6707 raft_consensus.cc:2932] T d3f2250a62c6429e886753aaffcbe883 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { replace: true } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false } } }
I20240501 07:59:13.434098 6165 catalog_manager.cc:5484] T d3f2250a62c6429e886753aaffcbe883 P 3b164cbbc00f4e8eb3254ddd349a3ff9 reported cstate change: config changed from index 7 to 8, a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 07:59:13.435976 11171 raft_consensus.cc:2932] T 7dd6566ccb6e43388146a6dfd1c4a88c P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false } } }
I20240501 07:59:13.437577 6159 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 7dd6566ccb6e43388146a6dfd1c4a88c with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 07:59:13.438361 6577 consensus_queue.cc:237] T d3f2250a62c6429e886753aaffcbe883 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false } }
I20240501 07:59:13.438289 6444 raft_consensus.cc:2932] T 7dd6566ccb6e43388146a6dfd1c4a88c P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false } } }
I20240501 07:59:13.439301 8925 raft_consensus.cc:2932] T 7dd6566ccb6e43388146a6dfd1c4a88c P 13d05ebdca8c4f489c597ce550dd773d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false } } }
I20240501 07:59:13.439797 9055 raft_consensus.cc:1261] T d3f2250a62c6429e886753aaffcbe883 P a505e7cb133c4392bfda8a4a20874ae9 [term 1 FOLLOWER]: Refusing update from remote peer 3b164cbbc00f4e8eb3254ddd349a3ff9: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 07:59:13.440165 10046 consensus_queue.cc:1035] T d3f2250a62c6429e886753aaffcbe883 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 07:59:13.440124 6167 catalog_manager.cc:5484] T 7dd6566ccb6e43388146a6dfd1c4a88c P 1f9cb72be98941b5a8e504cbe34132b9 reported cstate change: config changed from index 5 to 6, VOTER 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "1f9cb72be98941b5a8e504cbe34132b9" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 07:59:13.441246 9189 raft_consensus.cc:1261] T d3f2250a62c6429e886753aaffcbe883 P 83f20e84af4c40d09a4804b6b26e697f [term 1 FOLLOWER]: Refusing update from remote peer 3b164cbbc00f4e8eb3254ddd349a3ff9: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 07:59:13.442041 10046 consensus_queue.cc:1035] T d3f2250a62c6429e886753aaffcbe883 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 07:59:13.443608 6556 tablet_service.cc:1514] Processing DeleteTablet for tablet 7dd6566ccb6e43388146a6dfd1c4a88c with delete_type TABLET_DATA_TOMBSTONED (TS 3b164cbbc00f4e8eb3254ddd349a3ff9 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:53874
I20240501 07:59:13.445169 11758 tablet_replica.cc:330] T 7dd6566ccb6e43388146a6dfd1c4a88c P 3b164cbbc00f4e8eb3254ddd349a3ff9: stopping tablet replica
I20240501 07:59:13.445405 11758 raft_consensus.cc:2229] T 7dd6566ccb6e43388146a6dfd1c4a88c P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:13.445679 11758 raft_consensus.cc:2258] T 7dd6566ccb6e43388146a6dfd1c4a88c P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:13.445896 9189 raft_consensus.cc:2932] T d3f2250a62c6429e886753aaffcbe883 P 83f20e84af4c40d09a4804b6b26e697f [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false } } }
I20240501 07:59:13.445286 9507 raft_consensus.cc:2932] T d3f2250a62c6429e886753aaffcbe883 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false } } }
I20240501 07:59:13.446524 11758 ts_tablet_manager.cc:1875] T 7dd6566ccb6e43388146a6dfd1c4a88c P 3b164cbbc00f4e8eb3254ddd349a3ff9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:59:13.448777 6160 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d3f2250a62c6429e886753aaffcbe883 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 07:59:13.449198 9055 raft_consensus.cc:2932] T d3f2250a62c6429e886753aaffcbe883 P a505e7cb133c4392bfda8a4a20874ae9 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false } } }
I20240501 07:59:13.449487 6165 catalog_manager.cc:5484] T d3f2250a62c6429e886753aaffcbe883 P 3b164cbbc00f4e8eb3254ddd349a3ff9 reported cstate change: config changed from index 8 to 9, VOTER 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 07:59:13.453307 6687 tablet_service.cc:1514] Processing DeleteTablet for tablet d3f2250a62c6429e886753aaffcbe883 with delete_type TABLET_DATA_TOMBSTONED (TS 1f9cb72be98941b5a8e504cbe34132b9 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:38524
I20240501 07:59:13.454942 11758 ts_tablet_manager.cc:1888] T 7dd6566ccb6e43388146a6dfd1c4a88c P 3b164cbbc00f4e8eb3254ddd349a3ff9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 07:59:13.455097 11758 log.cc:1196] T 7dd6566ccb6e43388146a6dfd1c4a88c P 3b164cbbc00f4e8eb3254ddd349a3ff9: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/7dd6566ccb6e43388146a6dfd1c4a88c
I20240501 07:59:13.456091 6160 catalog_manager.cc:4830] TS 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131:34471): tablet 7dd6566ccb6e43388146a6dfd1c4a88c (table rebalance_test_table_17 [id=4de37f887eb14e2dbe5d1609df1e538f]) successfully deleted
I20240501 07:59:13.460055 11759 tablet_replica.cc:330] T d3f2250a62c6429e886753aaffcbe883 P 1f9cb72be98941b5a8e504cbe34132b9: stopping tablet replica
I20240501 07:59:13.460253 11759 raft_consensus.cc:2229] T d3f2250a62c6429e886753aaffcbe883 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:13.460472 11759 raft_consensus.cc:2258] T d3f2250a62c6429e886753aaffcbe883 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:13.461282 11759 ts_tablet_manager.cc:1875] T d3f2250a62c6429e886753aaffcbe883 P 1f9cb72be98941b5a8e504cbe34132b9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:59:13.463734 11759 ts_tablet_manager.cc:1888] T d3f2250a62c6429e886753aaffcbe883 P 1f9cb72be98941b5a8e504cbe34132b9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240501 07:59:13.463858 11759 log.cc:1196] T d3f2250a62c6429e886753aaffcbe883 P 1f9cb72be98941b5a8e504cbe34132b9: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/d3f2250a62c6429e886753aaffcbe883
I20240501 07:59:13.464442 6159 catalog_manager.cc:4830] TS 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132:36773): tablet d3f2250a62c6429e886753aaffcbe883 (table rebalance_test_table_13 [id=886e8147658c4cffb480ca3fb821c563]) successfully deleted
I20240501 07:59:13.473492 6444 consensus_queue.cc:237] T 6dc929e2a6ae45bd8b1f297583636bb6 P 8315d22c56f7439a9e1fc82f153f018c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: true } }
I20240501 07:59:13.475016 8793 raft_consensus.cc:1261] T 6dc929e2a6ae45bd8b1f297583636bb6 P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Refusing update from remote peer 8315d22c56f7439a9e1fc82f153f018c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 07:59:13.475061 6313 raft_consensus.cc:1261] T 6dc929e2a6ae45bd8b1f297583636bb6 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Refusing update from remote peer 8315d22c56f7439a9e1fc82f153f018c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 07:59:13.475462 10111 consensus_queue.cc:1035] T 6dc929e2a6ae45bd8b1f297583636bb6 P 8315d22c56f7439a9e1fc82f153f018c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 07:59:13.475652 11735 consensus_queue.cc:1035] T 6dc929e2a6ae45bd8b1f297583636bb6 P 8315d22c56f7439a9e1fc82f153f018c [LEADER]: Connected to new peer: Peer: permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240501 07:59:13.476127 6380 consensus_peers.cc:485] T 6dc929e2a6ae45bd8b1f297583636bb6 P 8315d22c56f7439a9e1fc82f153f018c -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6dc929e2a6ae45bd8b1f297583636bb6. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:13.477767 10111 raft_consensus.cc:2932] T 6dc929e2a6ae45bd8b1f297583636bb6 P 8315d22c56f7439a9e1fc82f153f018c [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: true } } }
I20240501 07:59:13.478477 6313 raft_consensus.cc:2932] T 6dc929e2a6ae45bd8b1f297583636bb6 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: true } } }
I20240501 07:59:13.479671 8793 raft_consensus.cc:2932] T 6dc929e2a6ae45bd8b1f297583636bb6 P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: true } } }
I20240501 07:59:13.480764 6166 catalog_manager.cc:5484] T 6dc929e2a6ae45bd8b1f297583636bb6 P 8315d22c56f7439a9e1fc82f153f018c reported cstate change: config changed from index 6 to 7, NON_VOTER 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "8315d22c56f7439a9e1fc82f153f018c" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 07:59:13.490141 6707 consensus_queue.cc:237] T 6daebcfc41754ec4aca939ca20588854 P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { promote: true } }
I20240501 07:59:13.491433 8925 raft_consensus.cc:1261] T 6daebcfc41754ec4aca939ca20588854 P 13d05ebdca8c4f489c597ce550dd773d [term 1 FOLLOWER]: Refusing update from remote peer 1f9cb72be98941b5a8e504cbe34132b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:13.491560 9055 raft_consensus.cc:1261] T 6daebcfc41754ec4aca939ca20588854 P a505e7cb133c4392bfda8a4a20874ae9 [term 1 FOLLOWER]: Refusing update from remote peer 1f9cb72be98941b5a8e504cbe34132b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 07:59:13.491747 6641 consensus_peers.cc:485] T 6daebcfc41754ec4aca939ca20588854 P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129:46875): Couldn't send request to peer 8b8b3ad3aa8f4053b2167dd550f2086f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6daebcfc41754ec4aca939ca20588854. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:13.491861 11171 consensus_queue.cc:1035] T 6daebcfc41754ec4aca939ca20588854 P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 07:59:13.492283 11171 consensus_queue.cc:1035] T 6daebcfc41754ec4aca939ca20588854 P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 07:59:13.493944 11171 raft_consensus.cc:2932] T 6daebcfc41754ec4aca939ca20588854 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { promote: true } } }
I20240501 07:59:13.494925 9055 raft_consensus.cc:2932] T 6daebcfc41754ec4aca939ca20588854 P a505e7cb133c4392bfda8a4a20874ae9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { promote: true } } }
I20240501 07:59:13.496258 8925 raft_consensus.cc:2932] T 6daebcfc41754ec4aca939ca20588854 P 13d05ebdca8c4f489c597ce550dd773d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { promote: true } } }
I20240501 07:59:13.497288 6165 catalog_manager.cc:5484] T 6daebcfc41754ec4aca939ca20588854 P 1f9cb72be98941b5a8e504cbe34132b9 reported cstate change: config changed from index -1 to 2, NON_VOTER 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129) added. New cstate: current_term: 1 leader_uuid: "1f9cb72be98941b5a8e504cbe34132b9" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 07:59:13.501667 9055 raft_consensus.cc:1203] T a756205204d146ce82a2799e30582f57 P a505e7cb133c4392bfda8a4a20874ae9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 07:59:13.536772 6167 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55950:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 07:59:13.541617 6313 raft_consensus.cc:1261] T 3484230eff9c4c25be777d8a0df76258 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Refusing update from remote peer 83f20e84af4c40d09a4804b6b26e697f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:13.541865 8793 raft_consensus.cc:1261] T 3484230eff9c4c25be777d8a0df76258 P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Refusing update from remote peer 83f20e84af4c40d09a4804b6b26e697f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:13.542119 11630 consensus_queue.cc:1035] T 3484230eff9c4c25be777d8a0df76258 P 83f20e84af4c40d09a4804b6b26e697f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:13.542419 11630 consensus_queue.cc:1035] T 3484230eff9c4c25be777d8a0df76258 P 83f20e84af4c40d09a4804b6b26e697f [LEADER]: Connected to new peer: Peer: permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:13.545974 6167 catalog_manager.cc:5576] TS 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133:39859) does not have the latest schema for tablet 3484230eff9c4c25be777d8a0df76258 (table rebalancer_dynamic_table [id=3abb7a980b184a8ea12a505903abf386]). Expected version 1 got 0
I20240501 07:59:13.547245 6172 catalog_manager.cc:5576] TS 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129:46875) does not have the latest schema for tablet 3484230eff9c4c25be777d8a0df76258 (table rebalancer_dynamic_table [id=3abb7a980b184a8ea12a505903abf386]). Expected version 1 got 0
I20240501 07:59:13.547847 11815 tablet.cc:1722] T 3484230eff9c4c25be777d8a0df76258 P 8b8b3ad3aa8f4053b2167dd550f2086f: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 07:59:13.548084 6165 catalog_manager.cc:5576] TS 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149) does not have the latest schema for tablet 3484230eff9c4c25be777d8a0df76258 (table rebalancer_dynamic_table [id=3abb7a980b184a8ea12a505903abf386]). Expected version 1 got 0
I20240501 07:59:13.548280 11815 tablet.cc:1620] T 3484230eff9c4c25be777d8a0df76258 P 8b8b3ad3aa8f4053b2167dd550f2086f: MemRowSet was empty: no flush needed.
I20240501 07:59:13.549556 11814 tablet.cc:1722] T 3484230eff9c4c25be777d8a0df76258 P 74077b61d6f54342bf9736b4a9f2eade: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 07:59:13.549868 11814 tablet.cc:1620] T 3484230eff9c4c25be777d8a0df76258 P 74077b61d6f54342bf9736b4a9f2eade: MemRowSet was empty: no flush needed.
I20240501 07:59:13.549983 6167 catalog_manager.cc:5576] TS 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133:39859) does not have the latest schema for tablet 3484230eff9c4c25be777d8a0df76258 (table rebalancer_dynamic_table [id=3abb7a980b184a8ea12a505903abf386]). Expected version 1 got 0
I20240501 07:59:13.549263 11813 tablet.cc:1722] T 3484230eff9c4c25be777d8a0df76258 P 83f20e84af4c40d09a4804b6b26e697f: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 07:59:13.550863 11813 tablet.cc:1620] T 3484230eff9c4c25be777d8a0df76258 P 83f20e84af4c40d09a4804b6b26e697f: MemRowSet was empty: no flush needed.
I20240501 07:59:13.551723 6165 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 0bc706e85ce74cef97a4827ea4d4aa65: rebalancer_dynamic_table [id=3abb7a980b184a8ea12a505903abf386] alter complete (version 1)
I20240501 07:59:13.585872 10945 consensus_queue.cc:1035] T e5bd3a8f7a7640c09774f781009eb19c P a505e7cb133c4392bfda8a4a20874ae9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:13.589807 11453 consensus_queue.cc:1035] T e5bd3a8f7a7640c09774f781009eb19c P a505e7cb133c4392bfda8a4a20874ae9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:13.610154 6577 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6e91683ff22243c1b3877733d74e2886"
dest_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:44098
I20240501 07:59:13.610312 6577 raft_consensus.cc:595] T 6e91683ff22243c1b3877733d74e2886 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 LEADER]: Received request to transfer leadership
I20240501 07:59:13.627702 8793 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "72aa4dc7664b48b7a73d34feba7d44d3"
dest_uuid: "74077b61d6f54342bf9736b4a9f2eade"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:56344
I20240501 07:59:13.627900 8793 raft_consensus.cc:595] T 72aa4dc7664b48b7a73d34feba7d44d3 P 74077b61d6f54342bf9736b4a9f2eade [term 1 LEADER]: Received request to transfer leadership
I20240501 07:59:13.635382 9055 raft_consensus.cc:1203] T d6c28cd91f624673b8e428f471a1dcce P a505e7cb133c4392bfda8a4a20874ae9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 07:59:13.638406 8925 consensus_queue.cc:237] T 542c9ac08f7d4bec9b15a309c2e21364 P 13d05ebdca8c4f489c597ce550dd773d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { replace: true } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: true } }
I20240501 07:59:13.639866 9055 raft_consensus.cc:1261] T 542c9ac08f7d4bec9b15a309c2e21364 P a505e7cb133c4392bfda8a4a20874ae9 [term 1 FOLLOWER]: Refusing update from remote peer 13d05ebdca8c4f489c597ce550dd773d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:13.640007 6707 raft_consensus.cc:1261] T 542c9ac08f7d4bec9b15a309c2e21364 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Refusing update from remote peer 13d05ebdca8c4f489c597ce550dd773d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 07:59:13.640105 8861 consensus_peers.cc:485] T 542c9ac08f7d4bec9b15a309c2e21364 P 13d05ebdca8c4f489c597ce550dd773d -> Peer 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131:34471): Couldn't send request to peer 3b164cbbc00f4e8eb3254ddd349a3ff9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 542c9ac08f7d4bec9b15a309c2e21364. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:13.640225 10478 consensus_queue.cc:1035] T 542c9ac08f7d4bec9b15a309c2e21364 P 13d05ebdca8c4f489c597ce550dd773d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 07:59:13.640509 9400 consensus_queue.cc:1035] T 542c9ac08f7d4bec9b15a309c2e21364 P 13d05ebdca8c4f489c597ce550dd773d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 07:59:13.642194 10478 raft_consensus.cc:2932] T 542c9ac08f7d4bec9b15a309c2e21364 P 13d05ebdca8c4f489c597ce550dd773d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { replace: true } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: true } } }
I20240501 07:59:13.642645 6707 raft_consensus.cc:2932] T 542c9ac08f7d4bec9b15a309c2e21364 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { replace: true } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: true } } }
I20240501 07:59:13.645576 11119 raft_consensus.cc:979] T 6e91683ff22243c1b3877733d74e2886 P 3b164cbbc00f4e8eb3254ddd349a3ff9: : Instructing follower 74077b61d6f54342bf9736b4a9f2eade to start an election
I20240501 07:59:13.645673 11119 raft_consensus.cc:1067] T 6e91683ff22243c1b3877733d74e2886 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 LEADER]: Signalling peer 74077b61d6f54342bf9736b4a9f2eade to start an election
I20240501 07:59:13.645998 6165 catalog_manager.cc:5484] T 542c9ac08f7d4bec9b15a309c2e21364 P 1f9cb72be98941b5a8e504cbe34132b9 reported cstate change: config changed from index -1 to 2, NON_VOTER 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131) added. New cstate: current_term: 1 leader_uuid: "13d05ebdca8c4f489c597ce550dd773d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { replace: true } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: true } } }
I20240501 07:59:13.647516 8793 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6e91683ff22243c1b3877733d74e2886"
dest_uuid: "74077b61d6f54342bf9736b4a9f2eade"
from {username='slave'} at 127.0.99.131:38735
I20240501 07:59:13.647679 8793 raft_consensus.cc:482] T 6e91683ff22243c1b3877733d74e2886 P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:59:13.647760 8793 raft_consensus.cc:3037] T 6e91683ff22243c1b3877733d74e2886 P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:13.648805 8793 raft_consensus.cc:504] T 6e91683ff22243c1b3877733d74e2886 P 74077b61d6f54342bf9736b4a9f2eade [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } }
I20240501 07:59:13.650449 8793 leader_election.cc:288] T 6e91683ff22243c1b3877733d74e2886 P 74077b61d6f54342bf9736b4a9f2eade [CANDIDATE]: Term 2 election: Requested vote from peers 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129:46875), 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131:34471), 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132:36773)
I20240501 07:59:13.650720 6311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e91683ff22243c1b3877733d74e2886" candidate_uuid: "74077b61d6f54342bf9736b4a9f2eade" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f"
I20240501 07:59:13.650840 6311 raft_consensus.cc:3037] T 6e91683ff22243c1b3877733d74e2886 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:13.651402 6707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e91683ff22243c1b3877733d74e2886" candidate_uuid: "74077b61d6f54342bf9736b4a9f2eade" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "1f9cb72be98941b5a8e504cbe34132b9"
I20240501 07:59:13.651525 6707 raft_consensus.cc:3037] T 6e91683ff22243c1b3877733d74e2886 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:13.651820 6311 raft_consensus.cc:2454] T 6e91683ff22243c1b3877733d74e2886 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74077b61d6f54342bf9736b4a9f2eade in term 2.
I20240501 07:59:13.652562 6707 raft_consensus.cc:2454] T 6e91683ff22243c1b3877733d74e2886 P 1f9cb72be98941b5a8e504cbe34132b9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74077b61d6f54342bf9736b4a9f2eade in term 2.
I20240501 07:59:13.653149 8729 leader_election.cc:302] T 6e91683ff22243c1b3877733d74e2886 P 74077b61d6f54342bf9736b4a9f2eade [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 1f9cb72be98941b5a8e504cbe34132b9, 74077b61d6f54342bf9736b4a9f2eade, 8b8b3ad3aa8f4053b2167dd550f2086f; no voters:
I20240501 07:59:13.653323 11723 raft_consensus.cc:2781] T 6e91683ff22243c1b3877733d74e2886 P 74077b61d6f54342bf9736b4a9f2eade [term 2 FOLLOWER]: Leader election won for term 2
I20240501 07:59:13.653404 11723 raft_consensus.cc:686] T 6e91683ff22243c1b3877733d74e2886 P 74077b61d6f54342bf9736b4a9f2eade [term 2 LEADER]: Becoming Leader. State: Replica: 74077b61d6f54342bf9736b4a9f2eade, State: Running, Role: LEADER
I20240501 07:59:13.653540 11723 consensus_queue.cc:237] T 6e91683ff22243c1b3877733d74e2886 P 74077b61d6f54342bf9736b4a9f2eade [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } }
I20240501 07:59:13.652897 6577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e91683ff22243c1b3877733d74e2886" candidate_uuid: "74077b61d6f54342bf9736b4a9f2eade" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9"
I20240501 07:59:13.654951 6577 raft_consensus.cc:3032] T 6e91683ff22243c1b3877733d74e2886 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 07:59:13.655002 6577 raft_consensus.cc:729] T 6e91683ff22243c1b3877733d74e2886 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3b164cbbc00f4e8eb3254ddd349a3ff9, State: Running, Role: LEADER
I20240501 07:59:13.655004 6444 consensus_queue.cc:237] T 54b3efabad7f4b608cc9d931546b5743 P 8315d22c56f7439a9e1fc82f153f018c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { replace: true } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { promote: true } }
I20240501 07:59:13.655139 6577 consensus_queue.cc:260] T 6e91683ff22243c1b3877733d74e2886 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } }
I20240501 07:59:13.655336 6577 raft_consensus.cc:3037] T 6e91683ff22243c1b3877733d74e2886 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:13.656381 6577 raft_consensus.cc:2454] T 6e91683ff22243c1b3877733d74e2886 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74077b61d6f54342bf9736b4a9f2eade in term 2.
I20240501 07:59:13.656513 9189 raft_consensus.cc:1261] T 54b3efabad7f4b608cc9d931546b5743 P 83f20e84af4c40d09a4804b6b26e697f [term 1 FOLLOWER]: Refusing update from remote peer 8315d22c56f7439a9e1fc82f153f018c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 07:59:13.656810 6379 consensus_peers.cc:485] T 54b3efabad7f4b608cc9d931546b5743 P 8315d22c56f7439a9e1fc82f153f018c -> Peer 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129:46875): Couldn't send request to peer 8b8b3ad3aa8f4053b2167dd550f2086f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 54b3efabad7f4b608cc9d931546b5743. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:13.657259 11735 consensus_queue.cc:1035] T 54b3efabad7f4b608cc9d931546b5743 P 8315d22c56f7439a9e1fc82f153f018c [LEADER]: Connected to new peer: Peer: permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 07:59:13.657644 6167 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55950:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 07:59:13.658743 9055 raft_consensus.cc:2932] T 542c9ac08f7d4bec9b15a309c2e21364 P a505e7cb133c4392bfda8a4a20874ae9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { replace: true } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: true } } }
I20240501 07:59:13.658996 6707 raft_consensus.cc:1261] T 54b3efabad7f4b608cc9d931546b5743 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Refusing update from remote peer 8315d22c56f7439a9e1fc82f153f018c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:13.659379 11735 consensus_queue.cc:1035] T 54b3efabad7f4b608cc9d931546b5743 P 8315d22c56f7439a9e1fc82f153f018c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 07:59:13.661612 6165 catalog_manager.cc:5484] T 6e91683ff22243c1b3877733d74e2886 P 74077b61d6f54342bf9736b4a9f2eade reported cstate change: term changed from 1 to 2, leader changed from 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131) to 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133). New cstate: current_term: 2 leader_uuid: "74077b61d6f54342bf9736b4a9f2eade" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 07:59:13.661847 8512 raft_consensus.cc:2932] T 54b3efabad7f4b608cc9d931546b5743 P 8315d22c56f7439a9e1fc82f153f018c [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { replace: true } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { promote: true } } }
I20240501 07:59:13.662302 9189 raft_consensus.cc:2932] T 54b3efabad7f4b608cc9d931546b5743 P 83f20e84af4c40d09a4804b6b26e697f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { replace: true } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { promote: true } } }
I20240501 07:59:13.665268 6172 catalog_manager.cc:5484] T 54b3efabad7f4b608cc9d931546b5743 P 8315d22c56f7439a9e1fc82f153f018c reported cstate change: config changed from index -1 to 2, NON_VOTER 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129) added. New cstate: current_term: 1 leader_uuid: "8315d22c56f7439a9e1fc82f153f018c" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 07:59:13.672389 11813 tablet.cc:1722] T 3484230eff9c4c25be777d8a0df76258 P 83f20e84af4c40d09a4804b6b26e697f: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 07:59:13.672719 11813 tablet.cc:1620] T 3484230eff9c4c25be777d8a0df76258 P 83f20e84af4c40d09a4804b6b26e697f: MemRowSet was empty: no flush needed.
I20240501 07:59:13.673163 11815 tablet.cc:1722] T 3484230eff9c4c25be777d8a0df76258 P 8b8b3ad3aa8f4053b2167dd550f2086f: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 07:59:13.673450 11815 tablet.cc:1620] T 3484230eff9c4c25be777d8a0df76258 P 8b8b3ad3aa8f4053b2167dd550f2086f: MemRowSet was empty: no flush needed.
I20240501 07:59:13.677732 11814 tablet.cc:1722] T 3484230eff9c4c25be777d8a0df76258 P 74077b61d6f54342bf9736b4a9f2eade: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 07:59:13.678083 11814 tablet.cc:1620] T 3484230eff9c4c25be777d8a0df76258 P 74077b61d6f54342bf9736b4a9f2eade: MemRowSet was empty: no flush needed.
I20240501 07:59:13.679791 6311 consensus_queue.cc:237] T 2e540f389b2a4aef93a020f549d11b67 P 8b8b3ad3aa8f4053b2167dd550f2086f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: true } }
I20240501 07:59:13.681219 6165 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 0bc706e85ce74cef97a4827ea4d4aa65: rebalancer_dynamic_table [id=3abb7a980b184a8ea12a505903abf386] alter complete (version 2)
I20240501 07:59:13.681826 9189 raft_consensus.cc:1261] T 2e540f389b2a4aef93a020f549d11b67 P 83f20e84af4c40d09a4804b6b26e697f [term 1 FOLLOWER]: Refusing update from remote peer 8b8b3ad3aa8f4053b2167dd550f2086f: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240501 07:59:13.682178 6250 consensus_peers.cc:485] T 2e540f389b2a4aef93a020f549d11b67 P 8b8b3ad3aa8f4053b2167dd550f2086f -> Peer 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134:40141): Couldn't send request to peer 13d05ebdca8c4f489c597ce550dd773d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2e540f389b2a4aef93a020f549d11b67. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:13.682381 11094 consensus_queue.cc:1035] T 2e540f389b2a4aef93a020f549d11b67 P 8b8b3ad3aa8f4053b2167dd550f2086f [LEADER]: Connected to new peer: Peer: permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 07:59:13.683290 6577 raft_consensus.cc:1261] T 2e540f389b2a4aef93a020f549d11b67 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Refusing update from remote peer 8b8b3ad3aa8f4053b2167dd550f2086f: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 07:59:13.683698 11094 consensus_queue.cc:1035] T 2e540f389b2a4aef93a020f549d11b67 P 8b8b3ad3aa8f4053b2167dd550f2086f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 07:59:13.687943 6708 raft_consensus.cc:2932] T 54b3efabad7f4b608cc9d931546b5743 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { replace: true } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { promote: true } } }
I20240501 07:59:13.689721 11094 raft_consensus.cc:2932] T 2e540f389b2a4aef93a020f549d11b67 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: true } } }
I20240501 07:59:13.691596 9189 raft_consensus.cc:2932] T 2e540f389b2a4aef93a020f549d11b67 P 83f20e84af4c40d09a4804b6b26e697f [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: true } } }
I20240501 07:59:13.692188 6577 raft_consensus.cc:2932] T 2e540f389b2a4aef93a020f549d11b67 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: true } } }
I20240501 07:59:13.693490 6165 catalog_manager.cc:5484] T 2e540f389b2a4aef93a020f549d11b67 P 8b8b3ad3aa8f4053b2167dd550f2086f reported cstate change: config changed from index 6 to 7, NON_VOTER 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 07:59:13.698529 10945 consensus_queue.cc:1035] T 2560b1ec81ac4357b32c7a8ba9d72216 P a505e7cb133c4392bfda8a4a20874ae9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:13.707042 8793 raft_consensus.cc:3037] T 2560b1ec81ac4357b32c7a8ba9d72216 P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:13.709164 10945 consensus_queue.cc:1035] T 2560b1ec81ac4357b32c7a8ba9d72216 P a505e7cb133c4392bfda8a4a20874ae9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:13.736766 8793 raft_consensus.cc:1203] T dc550f54e8024d47b1955905e7858a1e P 74077b61d6f54342bf9736b4a9f2eade [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 07:59:13.738587 10478 consensus_queue.cc:1035] T e0583fa43994452ca7553cbf2309eba1 P 13d05ebdca8c4f489c597ce550dd773d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:13.743150 11215 consensus_queue.cc:1035] T e0583fa43994452ca7553cbf2309eba1 P 13d05ebdca8c4f489c597ce550dd773d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:13.747610 9189 raft_consensus.cc:1203] T ff9dc384dbac47dc9795617c64688943 P 83f20e84af4c40d09a4804b6b26e697f [term 2 LEARNER]: Deduplicated request from leader. Original: 2.7->[2.8-2.8] Dedup: 2.8->[]
I20240501 07:59:13.766170 11723 raft_consensus.cc:979] T 72aa4dc7664b48b7a73d34feba7d44d3 P 74077b61d6f54342bf9736b4a9f2eade: : Instructing follower a505e7cb133c4392bfda8a4a20874ae9 to start an election
I20240501 07:59:13.766309 11723 raft_consensus.cc:1067] T 72aa4dc7664b48b7a73d34feba7d44d3 P 74077b61d6f54342bf9736b4a9f2eade [term 1 LEADER]: Signalling peer a505e7cb133c4392bfda8a4a20874ae9 to start an election
I20240501 07:59:13.766887 9056 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "72aa4dc7664b48b7a73d34feba7d44d3"
dest_uuid: "a505e7cb133c4392bfda8a4a20874ae9"
from {username='slave'} at 127.0.99.133:58533
I20240501 07:59:13.767061 9056 raft_consensus.cc:482] T 72aa4dc7664b48b7a73d34feba7d44d3 P a505e7cb133c4392bfda8a4a20874ae9 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:59:13.767131 9056 raft_consensus.cc:3037] T 72aa4dc7664b48b7a73d34feba7d44d3 P a505e7cb133c4392bfda8a4a20874ae9 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:13.768280 9056 raft_consensus.cc:504] T 72aa4dc7664b48b7a73d34feba7d44d3 P a505e7cb133c4392bfda8a4a20874ae9 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { replace: true } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { promote: false } }
I20240501 07:59:13.768694 9056 leader_election.cc:288] T 72aa4dc7664b48b7a73d34feba7d44d3 P a505e7cb133c4392bfda8a4a20874ae9 [CANDIDATE]: Term 2 election: Requested vote from peers 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133:39859), 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134:40141), 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130:32941)
I20240501 07:59:13.768950 8793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72aa4dc7664b48b7a73d34feba7d44d3" candidate_uuid: "a505e7cb133c4392bfda8a4a20874ae9" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "74077b61d6f54342bf9736b4a9f2eade"
I20240501 07:59:13.769115 8793 raft_consensus.cc:3032] T 72aa4dc7664b48b7a73d34feba7d44d3 P 74077b61d6f54342bf9736b4a9f2eade [term 1 LEADER]: Stepping down as leader of term 1
I20240501 07:59:13.769161 8793 raft_consensus.cc:729] T 72aa4dc7664b48b7a73d34feba7d44d3 P 74077b61d6f54342bf9736b4a9f2eade [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 74077b61d6f54342bf9736b4a9f2eade, State: Running, Role: LEADER
I20240501 07:59:13.769168 8925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72aa4dc7664b48b7a73d34feba7d44d3" candidate_uuid: "a505e7cb133c4392bfda8a4a20874ae9" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "13d05ebdca8c4f489c597ce550dd773d"
I20240501 07:59:13.769299 8925 raft_consensus.cc:3037] T 72aa4dc7664b48b7a73d34feba7d44d3 P 13d05ebdca8c4f489c597ce550dd773d [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:13.769337 8793 consensus_queue.cc:260] T 72aa4dc7664b48b7a73d34feba7d44d3 P 74077b61d6f54342bf9736b4a9f2eade [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { replace: true } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { promote: false } }
I20240501 07:59:13.769459 6444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72aa4dc7664b48b7a73d34feba7d44d3" candidate_uuid: "a505e7cb133c4392bfda8a4a20874ae9" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "8315d22c56f7439a9e1fc82f153f018c"
I20240501 07:59:13.769533 8793 raft_consensus.cc:3037] T 72aa4dc7664b48b7a73d34feba7d44d3 P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:13.769573 6444 raft_consensus.cc:3037] T 72aa4dc7664b48b7a73d34feba7d44d3 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:13.770327 6444 raft_consensus.cc:2454] T 72aa4dc7664b48b7a73d34feba7d44d3 P 8315d22c56f7439a9e1fc82f153f018c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a505e7cb133c4392bfda8a4a20874ae9 in term 2.
I20240501 07:59:13.770347 8925 raft_consensus.cc:2454] T 72aa4dc7664b48b7a73d34feba7d44d3 P 13d05ebdca8c4f489c597ce550dd773d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a505e7cb133c4392bfda8a4a20874ae9 in term 2.
I20240501 07:59:13.770498 8793 raft_consensus.cc:2454] T 72aa4dc7664b48b7a73d34feba7d44d3 P 74077b61d6f54342bf9736b4a9f2eade [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a505e7cb133c4392bfda8a4a20874ae9 in term 2.
I20240501 07:59:13.770717 8991 leader_election.cc:302] T 72aa4dc7664b48b7a73d34feba7d44d3 P a505e7cb133c4392bfda8a4a20874ae9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 13d05ebdca8c4f489c597ce550dd773d, 8315d22c56f7439a9e1fc82f153f018c, a505e7cb133c4392bfda8a4a20874ae9; no voters:
I20240501 07:59:13.771039 10945 raft_consensus.cc:2781] T 72aa4dc7664b48b7a73d34feba7d44d3 P a505e7cb133c4392bfda8a4a20874ae9 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 07:59:13.771126 10945 raft_consensus.cc:686] T 72aa4dc7664b48b7a73d34feba7d44d3 P a505e7cb133c4392bfda8a4a20874ae9 [term 2 LEADER]: Becoming Leader. State: Replica: a505e7cb133c4392bfda8a4a20874ae9, State: Running, Role: LEADER
I20240501 07:59:13.771268 10945 consensus_queue.cc:237] T 72aa4dc7664b48b7a73d34feba7d44d3 P a505e7cb133c4392bfda8a4a20874ae9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { replace: true } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { promote: false } }
I20240501 07:59:13.773367 6165 catalog_manager.cc:5484] T 72aa4dc7664b48b7a73d34feba7d44d3 P a505e7cb133c4392bfda8a4a20874ae9 reported cstate change: term changed from 1 to 2, leader changed from 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133) to a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135). New cstate: current_term: 2 leader_uuid: "a505e7cb133c4392bfda8a4a20874ae9" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 07:59:13.784571 6165 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55950:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 07:59:13.790402 11813 tablet.cc:1722] T 3484230eff9c4c25be777d8a0df76258 P 83f20e84af4c40d09a4804b6b26e697f: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 07:59:13.790676 11814 tablet.cc:1722] T 3484230eff9c4c25be777d8a0df76258 P 74077b61d6f54342bf9736b4a9f2eade: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 07:59:13.791688 11815 tablet.cc:1722] T 3484230eff9c4c25be777d8a0df76258 P 8b8b3ad3aa8f4053b2167dd550f2086f: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 07:59:13.796137 6165 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 0bc706e85ce74cef97a4827ea4d4aa65: rebalancer_dynamic_table_new_name [id=3abb7a980b184a8ea12a505903abf386] alter complete (version 3)
I20240501 07:59:13.807235 6444 raft_consensus.cc:1203] T f88fe033bce14bf0b1b63a52a7ba137e P 8315d22c56f7439a9e1fc82f153f018c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 07:59:13.821899 6165 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55950:
name: "rebalancer_extra_table_16"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 07:59:13.828012 9169 tablet_service.cc:1467] Processing CreateTablet for tablet a29720541b7b4a79aaa810592ef7c4b1 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=e1f47bf2d1a445e1ad4606f07f6ec24a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:13.828361 9169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a29720541b7b4a79aaa810592ef7c4b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:13.828466 6295 tablet_service.cc:1467] Processing CreateTablet for tablet 1c32cd00bc344ae1b161d3f9b43b53d3 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=e1f47bf2d1a445e1ad4606f07f6ec24a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:13.828734 6295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c32cd00bc344ae1b161d3f9b43b53d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:13.829438 8905 tablet_service.cc:1467] Processing CreateTablet for tablet a29720541b7b4a79aaa810592ef7c4b1 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=e1f47bf2d1a445e1ad4606f07f6ec24a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:13.829694 8905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a29720541b7b4a79aaa810592ef7c4b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:13.830106 6686 tablet_service.cc:1467] Processing CreateTablet for tablet 1c32cd00bc344ae1b161d3f9b43b53d3 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=e1f47bf2d1a445e1ad4606f07f6ec24a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:13.830389 6686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c32cd00bc344ae1b161d3f9b43b53d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:13.831068 6556 tablet_service.cc:1467] Processing CreateTablet for tablet 144ff11d2a504f7ab7a230e6822d24ca (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=e1f47bf2d1a445e1ad4606f07f6ec24a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:13.831326 6556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 144ff11d2a504f7ab7a230e6822d24ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:13.831825 11987 tablet_bootstrap.cc:492] T a29720541b7b4a79aaa810592ef7c4b1 P 13d05ebdca8c4f489c597ce550dd773d: Bootstrap starting.
I20240501 07:59:13.832692 11987 tablet_bootstrap.cc:654] T a29720541b7b4a79aaa810592ef7c4b1 P 13d05ebdca8c4f489c597ce550dd773d: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:13.833698 6294 tablet_service.cc:1467] Processing CreateTablet for tablet 144ff11d2a504f7ab7a230e6822d24ca (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=e1f47bf2d1a445e1ad4606f07f6ec24a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:13.833931 6294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 144ff11d2a504f7ab7a230e6822d24ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:13.834203 6687 tablet_service.cc:1467] Processing CreateTablet for tablet a29720541b7b4a79aaa810592ef7c4b1 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=e1f47bf2d1a445e1ad4606f07f6ec24a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:13.834452 6687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a29720541b7b4a79aaa810592ef7c4b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:13.835975 11990 tablet_bootstrap.cc:492] T 144ff11d2a504f7ab7a230e6822d24ca P 3b164cbbc00f4e8eb3254ddd349a3ff9: Bootstrap starting.
I20240501 07:59:13.836830 11990 tablet_bootstrap.cc:654] T 144ff11d2a504f7ab7a230e6822d24ca P 3b164cbbc00f4e8eb3254ddd349a3ff9: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:13.837858 11990 tablet_bootstrap.cc:492] T 144ff11d2a504f7ab7a230e6822d24ca P 3b164cbbc00f4e8eb3254ddd349a3ff9: No bootstrap required, opened a new log
I20240501 07:59:13.837947 11990 ts_tablet_manager.cc:1367] T 144ff11d2a504f7ab7a230e6822d24ca P 3b164cbbc00f4e8eb3254ddd349a3ff9: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 07:59:13.838147 9167 tablet_service.cc:1467] Processing CreateTablet for tablet 144ff11d2a504f7ab7a230e6822d24ca (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=e1f47bf2d1a445e1ad4606f07f6ec24a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:13.838276 11988 tablet_bootstrap.cc:492] T 1c32cd00bc344ae1b161d3f9b43b53d3 P 1f9cb72be98941b5a8e504cbe34132b9: Bootstrap starting.
I20240501 07:59:13.838399 9167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 144ff11d2a504f7ab7a230e6822d24ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:13.838425 11990 raft_consensus.cc:348] T 144ff11d2a504f7ab7a230e6822d24ca P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } }
I20240501 07:59:13.838523 11990 raft_consensus.cc:374] T 144ff11d2a504f7ab7a230e6822d24ca P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:13.838551 11990 raft_consensus.cc:729] T 144ff11d2a504f7ab7a230e6822d24ca P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b164cbbc00f4e8eb3254ddd349a3ff9, State: Initialized, Role: FOLLOWER
I20240501 07:59:13.838672 11990 consensus_queue.cc:260] T 144ff11d2a504f7ab7a230e6822d24ca P 3b164cbbc00f4e8eb3254ddd349a3ff9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } }
I20240501 07:59:13.838876 9169 tablet_service.cc:1467] Processing CreateTablet for tablet 1c32cd00bc344ae1b161d3f9b43b53d3 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=e1f47bf2d1a445e1ad4606f07f6ec24a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:13.838950 11990 ts_tablet_manager.cc:1398] T 144ff11d2a504f7ab7a230e6822d24ca P 3b164cbbc00f4e8eb3254ddd349a3ff9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:13.839567 11988 tablet_bootstrap.cc:654] T 1c32cd00bc344ae1b161d3f9b43b53d3 P 1f9cb72be98941b5a8e504cbe34132b9: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:13.840322 11991 tablet_bootstrap.cc:492] T 1c32cd00bc344ae1b161d3f9b43b53d3 P 8b8b3ad3aa8f4053b2167dd550f2086f: Bootstrap starting.
I20240501 07:59:13.840945 9169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c32cd00bc344ae1b161d3f9b43b53d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:13.841645 11988 tablet_bootstrap.cc:492] T 1c32cd00bc344ae1b161d3f9b43b53d3 P 1f9cb72be98941b5a8e504cbe34132b9: No bootstrap required, opened a new log
I20240501 07:59:13.841747 11988 ts_tablet_manager.cc:1367] T 1c32cd00bc344ae1b161d3f9b43b53d3 P 1f9cb72be98941b5a8e504cbe34132b9: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 07:59:13.842833 11991 tablet_bootstrap.cc:654] T 1c32cd00bc344ae1b161d3f9b43b53d3 P 8b8b3ad3aa8f4053b2167dd550f2086f: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:13.842236 11988 raft_consensus.cc:348] T 1c32cd00bc344ae1b161d3f9b43b53d3 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } }
I20240501 07:59:13.843536 11988 raft_consensus.cc:374] T 1c32cd00bc344ae1b161d3f9b43b53d3 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:13.843729 11988 raft_consensus.cc:729] T 1c32cd00bc344ae1b161d3f9b43b53d3 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f9cb72be98941b5a8e504cbe34132b9, State: Initialized, Role: FOLLOWER
I20240501 07:59:13.844098 11988 consensus_queue.cc:260] T 1c32cd00bc344ae1b161d3f9b43b53d3 P 1f9cb72be98941b5a8e504cbe34132b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } }
I20240501 07:59:13.844412 11988 ts_tablet_manager.cc:1398] T 1c32cd00bc344ae1b161d3f9b43b53d3 P 1f9cb72be98941b5a8e504cbe34132b9: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 07:59:13.844836 11986 tablet_bootstrap.cc:492] T a29720541b7b4a79aaa810592ef7c4b1 P 83f20e84af4c40d09a4804b6b26e697f: Bootstrap starting.
I20240501 07:59:13.845049 11988 tablet_bootstrap.cc:492] T a29720541b7b4a79aaa810592ef7c4b1 P 1f9cb72be98941b5a8e504cbe34132b9: Bootstrap starting.
I20240501 07:59:13.845935 11988 tablet_bootstrap.cc:654] T a29720541b7b4a79aaa810592ef7c4b1 P 1f9cb72be98941b5a8e504cbe34132b9: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:13.846186 11986 tablet_bootstrap.cc:654] T a29720541b7b4a79aaa810592ef7c4b1 P 83f20e84af4c40d09a4804b6b26e697f: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:13.844892 11991 tablet_bootstrap.cc:492] T 1c32cd00bc344ae1b161d3f9b43b53d3 P 8b8b3ad3aa8f4053b2167dd550f2086f: No bootstrap required, opened a new log
I20240501 07:59:13.846438 11991 ts_tablet_manager.cc:1367] T 1c32cd00bc344ae1b161d3f9b43b53d3 P 8b8b3ad3aa8f4053b2167dd550f2086f: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 07:59:13.847905 11986 tablet_bootstrap.cc:492] T a29720541b7b4a79aaa810592ef7c4b1 P 83f20e84af4c40d09a4804b6b26e697f: No bootstrap required, opened a new log
I20240501 07:59:13.848011 11986 ts_tablet_manager.cc:1367] T a29720541b7b4a79aaa810592ef7c4b1 P 83f20e84af4c40d09a4804b6b26e697f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 07:59:13.848410 11991 raft_consensus.cc:348] T 1c32cd00bc344ae1b161d3f9b43b53d3 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } }
I20240501 07:59:13.848450 11986 raft_consensus.cc:348] T a29720541b7b4a79aaa810592ef7c4b1 P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:59:13.848543 11986 raft_consensus.cc:374] T a29720541b7b4a79aaa810592ef7c4b1 P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:13.848544 11991 raft_consensus.cc:374] T 1c32cd00bc344ae1b161d3f9b43b53d3 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:13.848584 11986 raft_consensus.cc:729] T a29720541b7b4a79aaa810592ef7c4b1 P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83f20e84af4c40d09a4804b6b26e697f, State: Initialized, Role: FOLLOWER
I20240501 07:59:13.848584 11991 raft_consensus.cc:729] T 1c32cd00bc344ae1b161d3f9b43b53d3 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b8b3ad3aa8f4053b2167dd550f2086f, State: Initialized, Role: FOLLOWER
I20240501 07:59:13.848711 11991 consensus_queue.cc:260] T 1c32cd00bc344ae1b161d3f9b43b53d3 P 8b8b3ad3aa8f4053b2167dd550f2086f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } }
I20240501 07:59:13.848709 11986 consensus_queue.cc:260] T a29720541b7b4a79aaa810592ef7c4b1 P 83f20e84af4c40d09a4804b6b26e697f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:59:13.849007 11991 ts_tablet_manager.cc:1398] T 1c32cd00bc344ae1b161d3f9b43b53d3 P 8b8b3ad3aa8f4053b2167dd550f2086f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:59:13.849045 11986 ts_tablet_manager.cc:1398] T a29720541b7b4a79aaa810592ef7c4b1 P 83f20e84af4c40d09a4804b6b26e697f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:13.849188 11986 tablet_bootstrap.cc:492] T 144ff11d2a504f7ab7a230e6822d24ca P 83f20e84af4c40d09a4804b6b26e697f: Bootstrap starting.
I20240501 07:59:13.849189 11991 tablet_bootstrap.cc:492] T 144ff11d2a504f7ab7a230e6822d24ca P 8b8b3ad3aa8f4053b2167dd550f2086f: Bootstrap starting.
I20240501 07:59:13.849712 11987 tablet_bootstrap.cc:492] T a29720541b7b4a79aaa810592ef7c4b1 P 13d05ebdca8c4f489c597ce550dd773d: No bootstrap required, opened a new log
I20240501 07:59:13.849795 11987 ts_tablet_manager.cc:1367] T a29720541b7b4a79aaa810592ef7c4b1 P 13d05ebdca8c4f489c597ce550dd773d: Time spent bootstrapping tablet: real 0.018s user 0.001s sys 0.000s
I20240501 07:59:13.850122 11991 tablet_bootstrap.cc:654] T 144ff11d2a504f7ab7a230e6822d24ca P 8b8b3ad3aa8f4053b2167dd550f2086f: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:13.850122 11986 tablet_bootstrap.cc:654] T 144ff11d2a504f7ab7a230e6822d24ca P 83f20e84af4c40d09a4804b6b26e697f: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:13.850221 11987 raft_consensus.cc:348] T a29720541b7b4a79aaa810592ef7c4b1 P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:59:13.850327 11987 raft_consensus.cc:374] T a29720541b7b4a79aaa810592ef7c4b1 P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:13.850353 11987 raft_consensus.cc:729] T a29720541b7b4a79aaa810592ef7c4b1 P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13d05ebdca8c4f489c597ce550dd773d, State: Initialized, Role: FOLLOWER
I20240501 07:59:13.850474 11987 consensus_queue.cc:260] T a29720541b7b4a79aaa810592ef7c4b1 P 13d05ebdca8c4f489c597ce550dd773d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:59:13.850732 11987 ts_tablet_manager.cc:1398] T a29720541b7b4a79aaa810592ef7c4b1 P 13d05ebdca8c4f489c597ce550dd773d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:13.853649 11986 tablet_bootstrap.cc:492] T 144ff11d2a504f7ab7a230e6822d24ca P 83f20e84af4c40d09a4804b6b26e697f: No bootstrap required, opened a new log
I20240501 07:59:13.853744 11986 ts_tablet_manager.cc:1367] T 144ff11d2a504f7ab7a230e6822d24ca P 83f20e84af4c40d09a4804b6b26e697f: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 07:59:13.854207 11986 raft_consensus.cc:348] T 144ff11d2a504f7ab7a230e6822d24ca P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } }
I20240501 07:59:13.854316 11986 raft_consensus.cc:374] T 144ff11d2a504f7ab7a230e6822d24ca P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:13.854353 11986 raft_consensus.cc:729] T 144ff11d2a504f7ab7a230e6822d24ca P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83f20e84af4c40d09a4804b6b26e697f, State: Initialized, Role: FOLLOWER
I20240501 07:59:13.854508 11986 consensus_queue.cc:260] T 144ff11d2a504f7ab7a230e6822d24ca P 83f20e84af4c40d09a4804b6b26e697f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } }
I20240501 07:59:13.854801 11986 ts_tablet_manager.cc:1398] T 144ff11d2a504f7ab7a230e6822d24ca P 83f20e84af4c40d09a4804b6b26e697f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:13.854976 11986 tablet_bootstrap.cc:492] T 1c32cd00bc344ae1b161d3f9b43b53d3 P 83f20e84af4c40d09a4804b6b26e697f: Bootstrap starting.
I20240501 07:59:13.855016 11998 ts_tablet_manager.cc:902] T 6dc929e2a6ae45bd8b1f297583636bb6 P 83f20e84af4c40d09a4804b6b26e697f: Initiating tablet copy from peer 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130:32941)
I20240501 07:59:13.855549 11998 tablet_copy_client.cc:308] T 6dc929e2a6ae45bd8b1f297583636bb6 P 83f20e84af4c40d09a4804b6b26e697f: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:32941
I20240501 07:59:13.856024 6466 tablet_copy_service.cc:140] P 8315d22c56f7439a9e1fc82f153f018c: Received BeginTabletCopySession request for tablet 6dc929e2a6ae45bd8b1f297583636bb6 from peer 83f20e84af4c40d09a4804b6b26e697f ({username='slave'} at 127.0.99.136:51357)
I20240501 07:59:13.856135 6466 tablet_copy_service.cc:161] P 8315d22c56f7439a9e1fc82f153f018c: Beginning new tablet copy session on tablet 6dc929e2a6ae45bd8b1f297583636bb6 from peer 83f20e84af4c40d09a4804b6b26e697f at {username='slave'} at 127.0.99.136:51357: session id = 83f20e84af4c40d09a4804b6b26e697f-6dc929e2a6ae45bd8b1f297583636bb6
I20240501 07:59:13.856202 11986 tablet_bootstrap.cc:654] T 1c32cd00bc344ae1b161d3f9b43b53d3 P 83f20e84af4c40d09a4804b6b26e697f: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:13.857149 6466 tablet_copy_source_session.cc:205] T 6dc929e2a6ae45bd8b1f297583636bb6 P 8315d22c56f7439a9e1fc82f153f018c: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:13.858436 11988 tablet_bootstrap.cc:492] T a29720541b7b4a79aaa810592ef7c4b1 P 1f9cb72be98941b5a8e504cbe34132b9: No bootstrap required, opened a new log
I20240501 07:59:13.858528 11988 ts_tablet_manager.cc:1367] T a29720541b7b4a79aaa810592ef7c4b1 P 1f9cb72be98941b5a8e504cbe34132b9: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.002s
I20240501 07:59:13.859752 11988 raft_consensus.cc:348] T a29720541b7b4a79aaa810592ef7c4b1 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:59:13.859898 11988 raft_consensus.cc:374] T a29720541b7b4a79aaa810592ef7c4b1 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:13.859977 11988 raft_consensus.cc:729] T a29720541b7b4a79aaa810592ef7c4b1 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f9cb72be98941b5a8e504cbe34132b9, State: Initialized, Role: FOLLOWER
I20240501 07:59:13.860110 11988 consensus_queue.cc:260] T a29720541b7b4a79aaa810592ef7c4b1 P 1f9cb72be98941b5a8e504cbe34132b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:59:13.860519 11988 ts_tablet_manager.cc:1398] T a29720541b7b4a79aaa810592ef7c4b1 P 1f9cb72be98941b5a8e504cbe34132b9: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 07:59:13.860877 11991 tablet_bootstrap.cc:492] T 144ff11d2a504f7ab7a230e6822d24ca P 8b8b3ad3aa8f4053b2167dd550f2086f: No bootstrap required, opened a new log
I20240501 07:59:13.861013 11991 ts_tablet_manager.cc:1367] T 144ff11d2a504f7ab7a230e6822d24ca P 8b8b3ad3aa8f4053b2167dd550f2086f: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240501 07:59:13.861518 11991 raft_consensus.cc:348] T 144ff11d2a504f7ab7a230e6822d24ca P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } }
I20240501 07:59:13.861644 11991 raft_consensus.cc:374] T 144ff11d2a504f7ab7a230e6822d24ca P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:13.861735 11991 raft_consensus.cc:729] T 144ff11d2a504f7ab7a230e6822d24ca P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b8b3ad3aa8f4053b2167dd550f2086f, State: Initialized, Role: FOLLOWER
I20240501 07:59:13.861933 11991 consensus_queue.cc:260] T 144ff11d2a504f7ab7a230e6822d24ca P 8b8b3ad3aa8f4053b2167dd550f2086f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } }
I20240501 07:59:13.860857 11998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6dc929e2a6ae45bd8b1f297583636bb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:13.863145 11991 ts_tablet_manager.cc:1398] T 144ff11d2a504f7ab7a230e6822d24ca P 8b8b3ad3aa8f4053b2167dd550f2086f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:59:13.866835 11998 tablet_copy_client.cc:783] T 6dc929e2a6ae45bd8b1f297583636bb6 P 83f20e84af4c40d09a4804b6b26e697f: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:13.867005 11998 tablet_copy_client.cc:647] T 6dc929e2a6ae45bd8b1f297583636bb6 P 83f20e84af4c40d09a4804b6b26e697f: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:13.868935 11998 tablet_copy_client.cc:523] T 6dc929e2a6ae45bd8b1f297583636bb6 P 83f20e84af4c40d09a4804b6b26e697f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:13.869520 11986 tablet_bootstrap.cc:492] T 1c32cd00bc344ae1b161d3f9b43b53d3 P 83f20e84af4c40d09a4804b6b26e697f: No bootstrap required, opened a new log
I20240501 07:59:13.869622 11986 ts_tablet_manager.cc:1367] T 1c32cd00bc344ae1b161d3f9b43b53d3 P 83f20e84af4c40d09a4804b6b26e697f: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.000s
I20240501 07:59:13.870462 11998 tablet_bootstrap.cc:492] T 6dc929e2a6ae45bd8b1f297583636bb6 P 83f20e84af4c40d09a4804b6b26e697f: Bootstrap starting.
I20240501 07:59:13.871237 11986 raft_consensus.cc:348] T 1c32cd00bc344ae1b161d3f9b43b53d3 P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } }
I20240501 07:59:13.871338 11986 raft_consensus.cc:374] T 1c32cd00bc344ae1b161d3f9b43b53d3 P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:13.871373 11986 raft_consensus.cc:729] T 1c32cd00bc344ae1b161d3f9b43b53d3 P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83f20e84af4c40d09a4804b6b26e697f, State: Initialized, Role: FOLLOWER
I20240501 07:59:13.871836 11986 consensus_queue.cc:260] T 1c32cd00bc344ae1b161d3f9b43b53d3 P 83f20e84af4c40d09a4804b6b26e697f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } }
I20240501 07:59:13.873114 11986 ts_tablet_manager.cc:1398] T 1c32cd00bc344ae1b161d3f9b43b53d3 P 83f20e84af4c40d09a4804b6b26e697f: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 07:59:13.892122 11998 tablet_bootstrap.cc:492] T 6dc929e2a6ae45bd8b1f297583636bb6 P 83f20e84af4c40d09a4804b6b26e697f: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:13.892481 11998 tablet_bootstrap.cc:492] T 6dc929e2a6ae45bd8b1f297583636bb6 P 83f20e84af4c40d09a4804b6b26e697f: Bootstrap complete.
I20240501 07:59:13.892663 11998 ts_tablet_manager.cc:1367] T 6dc929e2a6ae45bd8b1f297583636bb6 P 83f20e84af4c40d09a4804b6b26e697f: Time spent bootstrapping tablet: real 0.022s user 0.017s sys 0.002s
I20240501 07:59:13.893138 11998 raft_consensus.cc:348] T 6dc929e2a6ae45bd8b1f297583636bb6 P 83f20e84af4c40d09a4804b6b26e697f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: true } }
I20240501 07:59:13.893276 11998 raft_consensus.cc:729] T 6dc929e2a6ae45bd8b1f297583636bb6 P 83f20e84af4c40d09a4804b6b26e697f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 83f20e84af4c40d09a4804b6b26e697f, State: Initialized, Role: LEARNER
I20240501 07:59:13.893389 11998 consensus_queue.cc:260] T 6dc929e2a6ae45bd8b1f297583636bb6 P 83f20e84af4c40d09a4804b6b26e697f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: true } }
I20240501 07:59:13.893692 11998 ts_tablet_manager.cc:1398] T 6dc929e2a6ae45bd8b1f297583636bb6 P 83f20e84af4c40d09a4804b6b26e697f: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240501 07:59:13.894238 6466 tablet_copy_service.cc:342] P 8315d22c56f7439a9e1fc82f153f018c: Request end of tablet copy session 83f20e84af4c40d09a4804b6b26e697f-6dc929e2a6ae45bd8b1f297583636bb6 received from {username='slave'} at 127.0.99.136:51357
I20240501 07:59:13.894327 6466 tablet_copy_service.cc:433] P 8315d22c56f7439a9e1fc82f153f018c: ending tablet copy session 83f20e84af4c40d09a4804b6b26e697f-6dc929e2a6ae45bd8b1f297583636bb6 on tablet 6dc929e2a6ae45bd8b1f297583636bb6 with peer 83f20e84af4c40d09a4804b6b26e697f
I20240501 07:59:13.897563 10324 raft_consensus.cc:482] T a29720541b7b4a79aaa810592ef7c4b1 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:13.897696 10324 raft_consensus.cc:504] T a29720541b7b4a79aaa810592ef7c4b1 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:59:13.898087 10324 leader_election.cc:288] T a29720541b7b4a79aaa810592ef7c4b1 P 1f9cb72be98941b5a8e504cbe34132b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149), 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134:40141)
I20240501 07:59:13.898391 9189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a29720541b7b4a79aaa810592ef7c4b1" candidate_uuid: "1f9cb72be98941b5a8e504cbe34132b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83f20e84af4c40d09a4804b6b26e697f" is_pre_election: true
I20240501 07:59:13.898523 9189 raft_consensus.cc:2454] T a29720541b7b4a79aaa810592ef7c4b1 P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f9cb72be98941b5a8e504cbe34132b9 in term 0.
I20240501 07:59:13.898564 8925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a29720541b7b4a79aaa810592ef7c4b1" candidate_uuid: "1f9cb72be98941b5a8e504cbe34132b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13d05ebdca8c4f489c597ce550dd773d" is_pre_election: true
I20240501 07:59:13.898557 6165 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55950:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 07:59:13.898682 6165 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55950:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 07:59:13.898705 8925 raft_consensus.cc:2454] T a29720541b7b4a79aaa810592ef7c4b1 P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f9cb72be98941b5a8e504cbe34132b9 in term 0.
I20240501 07:59:13.898787 6642 leader_election.cc:302] T a29720541b7b4a79aaa810592ef7c4b1 P 1f9cb72be98941b5a8e504cbe34132b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f9cb72be98941b5a8e504cbe34132b9, 83f20e84af4c40d09a4804b6b26e697f; no voters:
I20240501 07:59:13.898952 10324 raft_consensus.cc:2781] T a29720541b7b4a79aaa810592ef7c4b1 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:13.899013 10324 raft_consensus.cc:482] T a29720541b7b4a79aaa810592ef7c4b1 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:13.899061 10324 raft_consensus.cc:3037] T a29720541b7b4a79aaa810592ef7c4b1 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:13.900315 10324 raft_consensus.cc:504] T a29720541b7b4a79aaa810592ef7c4b1 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:59:13.900713 10324 leader_election.cc:288] T a29720541b7b4a79aaa810592ef7c4b1 P 1f9cb72be98941b5a8e504cbe34132b9 [CANDIDATE]: Term 1 election: Requested vote from peers 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149), 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134:40141)
I20240501 07:59:13.901108 8925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a29720541b7b4a79aaa810592ef7c4b1" candidate_uuid: "1f9cb72be98941b5a8e504cbe34132b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13d05ebdca8c4f489c597ce550dd773d"
I20240501 07:59:13.901162 6165 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 0bc706e85ce74cef97a4827ea4d4aa65: Sending DeleteTablet for 3 replicas of tablet 3484230eff9c4c25be777d8a0df76258
I20240501 07:59:13.901190 9189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a29720541b7b4a79aaa810592ef7c4b1" candidate_uuid: "1f9cb72be98941b5a8e504cbe34132b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83f20e84af4c40d09a4804b6b26e697f"
I20240501 07:59:13.901252 8925 raft_consensus.cc:3037] T a29720541b7b4a79aaa810592ef7c4b1 P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:13.901329 9189 raft_consensus.cc:3037] T a29720541b7b4a79aaa810592ef7c4b1 P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:13.901698 9169 tablet_service.cc:1514] Processing DeleteTablet for tablet 3484230eff9c4c25be777d8a0df76258 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:13 UTC) from {username='slave'} at 127.0.0.1:33594
I20240501 07:59:13.902043 12006 tablet_replica.cc:330] T 3484230eff9c4c25be777d8a0df76258 P 83f20e84af4c40d09a4804b6b26e697f: stopping tablet replica
I20240501 07:59:13.902315 8925 raft_consensus.cc:2454] T a29720541b7b4a79aaa810592ef7c4b1 P 13d05ebdca8c4f489c597ce550dd773d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f9cb72be98941b5a8e504cbe34132b9 in term 1.
I20240501 07:59:13.902284 12006 raft_consensus.cc:2229] T 3484230eff9c4c25be777d8a0df76258 P 83f20e84af4c40d09a4804b6b26e697f [term 1 LEADER]: Raft consensus shutting down.
I20240501 07:59:13.902310 9189 raft_consensus.cc:2454] T a29720541b7b4a79aaa810592ef7c4b1 P 83f20e84af4c40d09a4804b6b26e697f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f9cb72be98941b5a8e504cbe34132b9 in term 1.
I20240501 07:59:13.902604 12006 raft_consensus.cc:2258] T 3484230eff9c4c25be777d8a0df76258 P 83f20e84af4c40d09a4804b6b26e697f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:13.903138 8757 tablet_service.cc:1514] Processing DeleteTablet for tablet 3484230eff9c4c25be777d8a0df76258 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:13 UTC) from {username='slave'} at 127.0.0.1:43136
I20240501 07:59:13.903157 6643 leader_election.cc:302] T a29720541b7b4a79aaa810592ef7c4b1 P 1f9cb72be98941b5a8e504cbe34132b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13d05ebdca8c4f489c597ce550dd773d, 1f9cb72be98941b5a8e504cbe34132b9; no voters:
I20240501 07:59:13.903307 12006 ts_tablet_manager.cc:1875] T 3484230eff9c4c25be777d8a0df76258 P 83f20e84af4c40d09a4804b6b26e697f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:59:13.903379 6295 tablet_service.cc:1514] Processing DeleteTablet for tablet 3484230eff9c4c25be777d8a0df76258 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:13 UTC) from {username='slave'} at 127.0.0.1:45480
I20240501 07:59:13.903484 12007 tablet_replica.cc:330] T 3484230eff9c4c25be777d8a0df76258 P 74077b61d6f54342bf9736b4a9f2eade: stopping tablet replica
I20240501 07:59:13.903656 10324 raft_consensus.cc:2781] T a29720541b7b4a79aaa810592ef7c4b1 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:13.903707 12007 raft_consensus.cc:2229] T 3484230eff9c4c25be777d8a0df76258 P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:13.903738 10324 raft_consensus.cc:686] T a29720541b7b4a79aaa810592ef7c4b1 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 LEADER]: Becoming Leader. State: Replica: 1f9cb72be98941b5a8e504cbe34132b9, State: Running, Role: LEADER
I20240501 07:59:13.903836 12007 raft_consensus.cc:2258] T 3484230eff9c4c25be777d8a0df76258 P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:13.903869 10324 consensus_queue.cc:237] T a29720541b7b4a79aaa810592ef7c4b1 P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:59:13.904428 12007 ts_tablet_manager.cc:1875] T 3484230eff9c4c25be777d8a0df76258 P 74077b61d6f54342bf9736b4a9f2eade: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:59:13.905095 12008 tablet_replica.cc:330] T 3484230eff9c4c25be777d8a0df76258 P 8b8b3ad3aa8f4053b2167dd550f2086f: stopping tablet replica
I20240501 07:59:13.905596 12006 ts_tablet_manager.cc:1888] T 3484230eff9c4c25be777d8a0df76258 P 83f20e84af4c40d09a4804b6b26e697f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:59:13.905586 12008 raft_consensus.cc:2229] T 3484230eff9c4c25be777d8a0df76258 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:13.905722 12006 log.cc:1196] T 3484230eff9c4c25be777d8a0df76258 P 83f20e84af4c40d09a4804b6b26e697f: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/3484230eff9c4c25be777d8a0df76258
I20240501 07:59:13.905777 12008 raft_consensus.cc:2258] T 3484230eff9c4c25be777d8a0df76258 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:13.906008 12006 ts_tablet_manager.cc:1909] T 3484230eff9c4c25be777d8a0df76258 P 83f20e84af4c40d09a4804b6b26e697f: Deleting consensus metadata
I20240501 07:59:13.906185 6165 catalog_manager.cc:5484] T a29720541b7b4a79aaa810592ef7c4b1 P 1f9cb72be98941b5a8e504cbe34132b9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132). New cstate: current_term: 1 leader_uuid: "1f9cb72be98941b5a8e504cbe34132b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } health_report { overall_health: HEALTHY } } }
I20240501 07:59:13.906440 12008 ts_tablet_manager.cc:1875] T 3484230eff9c4c25be777d8a0df76258 P 8b8b3ad3aa8f4053b2167dd550f2086f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:59:13.906661 6158 catalog_manager.cc:4830] TS 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): tablet 3484230eff9c4c25be777d8a0df76258 (table rebalancer_dynamic_table_new_name [id=3abb7a980b184a8ea12a505903abf386]) successfully deleted
I20240501 07:59:13.909200 10111 raft_consensus.cc:1050] T df00e37199e944589a0a5f3feeee308a P 8315d22c56f7439a9e1fc82f153f018c: attempting to promote NON_VOTER 13d05ebdca8c4f489c597ce550dd773d to VOTER
I20240501 07:59:13.909503 12008 ts_tablet_manager.cc:1888] T 3484230eff9c4c25be777d8a0df76258 P 8b8b3ad3aa8f4053b2167dd550f2086f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:59:13.909590 12008 log.cc:1196] T 3484230eff9c4c25be777d8a0df76258 P 8b8b3ad3aa8f4053b2167dd550f2086f: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/3484230eff9c4c25be777d8a0df76258
I20240501 07:59:13.909557 10111 consensus_queue.cc:237] T df00e37199e944589a0a5f3feeee308a P 8315d22c56f7439a9e1fc82f153f018c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false } }
I20240501 07:59:13.909798 12007 ts_tablet_manager.cc:1888] T 3484230eff9c4c25be777d8a0df76258 P 74077b61d6f54342bf9736b4a9f2eade: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:59:13.909924 12007 log.cc:1196] T 3484230eff9c4c25be777d8a0df76258 P 74077b61d6f54342bf9736b4a9f2eade: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/3484230eff9c4c25be777d8a0df76258
I20240501 07:59:13.910013 12008 ts_tablet_manager.cc:1909] T 3484230eff9c4c25be777d8a0df76258 P 8b8b3ad3aa8f4053b2167dd550f2086f: Deleting consensus metadata
I20240501 07:59:13.910238 12007 ts_tablet_manager.cc:1909] T 3484230eff9c4c25be777d8a0df76258 P 74077b61d6f54342bf9736b4a9f2eade: Deleting consensus metadata
I20240501 07:59:13.910804 6157 catalog_manager.cc:4830] TS 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129:46875): tablet 3484230eff9c4c25be777d8a0df76258 (table rebalancer_dynamic_table_new_name [id=3abb7a980b184a8ea12a505903abf386]) successfully deleted
I20240501 07:59:13.911023 6577 raft_consensus.cc:1261] T df00e37199e944589a0a5f3feeee308a P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Refusing update from remote peer 8315d22c56f7439a9e1fc82f153f018c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:59:13.911404 6708 raft_consensus.cc:1261] T df00e37199e944589a0a5f3feeee308a P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Refusing update from remote peer 8315d22c56f7439a9e1fc82f153f018c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:59:13.911484 6157 catalog_manager.cc:4830] TS 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133:39859): tablet 3484230eff9c4c25be777d8a0df76258 (table rebalancer_dynamic_table_new_name [id=3abb7a980b184a8ea12a505903abf386]) successfully deleted
I20240501 07:59:13.911484 10111 consensus_queue.cc:1035] T df00e37199e944589a0a5f3feeee308a P 8315d22c56f7439a9e1fc82f153f018c [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:59:13.911818 8512 consensus_queue.cc:1035] T df00e37199e944589a0a5f3feeee308a P 8315d22c56f7439a9e1fc82f153f018c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:59:13.912082 8925 raft_consensus.cc:1261] T df00e37199e944589a0a5f3feeee308a P 13d05ebdca8c4f489c597ce550dd773d [term 1 LEARNER]: Refusing update from remote peer 8315d22c56f7439a9e1fc82f153f018c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:59:13.912894 12011 consensus_queue.cc:1035] T df00e37199e944589a0a5f3feeee308a P 8315d22c56f7439a9e1fc82f153f018c [LEADER]: Connected to new peer: Peer: permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:59:13.914481 8512 raft_consensus.cc:2932] T df00e37199e944589a0a5f3feeee308a P 8315d22c56f7439a9e1fc82f153f018c [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false } } }
I20240501 07:59:13.914975 6708 raft_consensus.cc:2932] T df00e37199e944589a0a5f3feeee308a P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false } } }
I20240501 07:59:13.916260 6577 raft_consensus.cc:2932] T df00e37199e944589a0a5f3feeee308a P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false } } }
I20240501 07:59:13.918049 6165 catalog_manager.cc:5484] T df00e37199e944589a0a5f3feeee308a P 1f9cb72be98941b5a8e504cbe34132b9 reported cstate change: config changed from index 4 to 5, 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8315d22c56f7439a9e1fc82f153f018c" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false } } }
I20240501 07:59:13.918097 8925 raft_consensus.cc:2932] T df00e37199e944589a0a5f3feeee308a P 13d05ebdca8c4f489c597ce550dd773d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false } } }
I20240501 07:59:13.923321 6444 consensus_queue.cc:237] T df00e37199e944589a0a5f3feeee308a P 8315d22c56f7439a9e1fc82f153f018c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false } }
I20240501 07:59:13.924566 8925 raft_consensus.cc:1261] T df00e37199e944589a0a5f3feeee308a P 13d05ebdca8c4f489c597ce550dd773d [term 1 FOLLOWER]: Refusing update from remote peer 8315d22c56f7439a9e1fc82f153f018c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 07:59:13.924566 6708 raft_consensus.cc:1261] T df00e37199e944589a0a5f3feeee308a P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Refusing update from remote peer 8315d22c56f7439a9e1fc82f153f018c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 07:59:13.924978 8512 consensus_queue.cc:1035] T df00e37199e944589a0a5f3feeee308a P 8315d22c56f7439a9e1fc82f153f018c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 07:59:13.925163 12011 consensus_queue.cc:1035] T df00e37199e944589a0a5f3feeee308a P 8315d22c56f7439a9e1fc82f153f018c [LEADER]: Connected to new peer: Peer: permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 07:59:13.927022 8512 raft_consensus.cc:2932] T df00e37199e944589a0a5f3feeee308a P 8315d22c56f7439a9e1fc82f153f018c [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false } } }
I20240501 07:59:13.927431 6708 raft_consensus.cc:2932] T df00e37199e944589a0a5f3feeee308a P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false } } }
I20240501 07:59:13.928087 8925 raft_consensus.cc:2932] T df00e37199e944589a0a5f3feeee308a P 13d05ebdca8c4f489c597ce550dd773d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false } } }
I20240501 07:59:13.929284 6159 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet df00e37199e944589a0a5f3feeee308a with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 07:59:13.930235 6171 catalog_manager.cc:5484] T df00e37199e944589a0a5f3feeee308a P 1f9cb72be98941b5a8e504cbe34132b9 reported cstate change: config changed from index 5 to 6, VOTER 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "8315d22c56f7439a9e1fc82f153f018c" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false } } }
I20240501 07:59:13.932945 6556 tablet_service.cc:1514] Processing DeleteTablet for tablet df00e37199e944589a0a5f3feeee308a with delete_type TABLET_DATA_TOMBSTONED (TS 3b164cbbc00f4e8eb3254ddd349a3ff9 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:53874
I20240501 07:59:13.933266 11758 tablet_replica.cc:330] T df00e37199e944589a0a5f3feeee308a P 3b164cbbc00f4e8eb3254ddd349a3ff9: stopping tablet replica
I20240501 07:59:13.933423 11758 raft_consensus.cc:2229] T df00e37199e944589a0a5f3feeee308a P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:13.933602 11758 raft_consensus.cc:2258] T df00e37199e944589a0a5f3feeee308a P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:13.934285 11758 ts_tablet_manager.cc:1875] T df00e37199e944589a0a5f3feeee308a P 3b164cbbc00f4e8eb3254ddd349a3ff9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:59:13.936013 11758 ts_tablet_manager.cc:1888] T df00e37199e944589a0a5f3feeee308a P 3b164cbbc00f4e8eb3254ddd349a3ff9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 07:59:13.936077 11758 log.cc:1196] T df00e37199e944589a0a5f3feeee308a P 3b164cbbc00f4e8eb3254ddd349a3ff9: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/df00e37199e944589a0a5f3feeee308a
I20240501 07:59:13.936538 6160 catalog_manager.cc:4830] TS 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131:34471): tablet df00e37199e944589a0a5f3feeee308a (table rebalance_test_table_5 [id=336b9f044a794b6a8a51d76e720b0ecd]) successfully deleted
I20240501 07:59:13.952188 10046 raft_consensus.cc:482] T 144ff11d2a504f7ab7a230e6822d24ca P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:13.952342 10046 raft_consensus.cc:504] T 144ff11d2a504f7ab7a230e6822d24ca P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } }
I20240501 07:59:13.952836 10046 leader_election.cc:288] T 144ff11d2a504f7ab7a230e6822d24ca P 3b164cbbc00f4e8eb3254ddd349a3ff9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129:46875), 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149)
I20240501 07:59:13.953115 11901 raft_consensus.cc:482] T 1c32cd00bc344ae1b161d3f9b43b53d3 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:13.953279 11901 raft_consensus.cc:504] T 1c32cd00bc344ae1b161d3f9b43b53d3 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } }
I20240501 07:59:13.953409 9189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "144ff11d2a504f7ab7a230e6822d24ca" candidate_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83f20e84af4c40d09a4804b6b26e697f" is_pre_election: true
I20240501 07:59:13.953428 6311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "144ff11d2a504f7ab7a230e6822d24ca" candidate_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" is_pre_election: true
I20240501 07:59:13.953534 9189 raft_consensus.cc:2454] T 144ff11d2a504f7ab7a230e6822d24ca P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b164cbbc00f4e8eb3254ddd349a3ff9 in term 0.
I20240501 07:59:13.953564 6311 raft_consensus.cc:2454] T 144ff11d2a504f7ab7a230e6822d24ca P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b164cbbc00f4e8eb3254ddd349a3ff9 in term 0.
I20240501 07:59:13.953817 11901 leader_election.cc:288] T 1c32cd00bc344ae1b161d3f9b43b53d3 P 8b8b3ad3aa8f4053b2167dd550f2086f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132:36773), 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149)
I20240501 07:59:13.953881 6511 leader_election.cc:302] T 144ff11d2a504f7ab7a230e6822d24ca P 3b164cbbc00f4e8eb3254ddd349a3ff9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b164cbbc00f4e8eb3254ddd349a3ff9, 83f20e84af4c40d09a4804b6b26e697f; no voters:
I20240501 07:59:13.954171 10046 raft_consensus.cc:2781] T 144ff11d2a504f7ab7a230e6822d24ca P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:13.954234 10046 raft_consensus.cc:482] T 144ff11d2a504f7ab7a230e6822d24ca P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:13.954221 9189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c32cd00bc344ae1b161d3f9b43b53d3" candidate_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83f20e84af4c40d09a4804b6b26e697f" is_pre_election: true
I20240501 07:59:13.954278 10046 raft_consensus.cc:3037] T 144ff11d2a504f7ab7a230e6822d24ca P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:13.954277 6708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c32cd00bc344ae1b161d3f9b43b53d3" candidate_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f9cb72be98941b5a8e504cbe34132b9" is_pre_election: true
I20240501 07:59:13.954344 9189 raft_consensus.cc:2454] T 1c32cd00bc344ae1b161d3f9b43b53d3 P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b8b3ad3aa8f4053b2167dd550f2086f in term 0.
I20240501 07:59:13.954438 6708 raft_consensus.cc:2454] T 1c32cd00bc344ae1b161d3f9b43b53d3 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b8b3ad3aa8f4053b2167dd550f2086f in term 0.
I20240501 07:59:13.954641 6249 leader_election.cc:302] T 1c32cd00bc344ae1b161d3f9b43b53d3 P 8b8b3ad3aa8f4053b2167dd550f2086f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83f20e84af4c40d09a4804b6b26e697f, 8b8b3ad3aa8f4053b2167dd550f2086f; no voters:
I20240501 07:59:13.954820 11901 raft_consensus.cc:2781] T 1c32cd00bc344ae1b161d3f9b43b53d3 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:13.954885 11901 raft_consensus.cc:482] T 1c32cd00bc344ae1b161d3f9b43b53d3 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:13.954921 11901 raft_consensus.cc:3037] T 1c32cd00bc344ae1b161d3f9b43b53d3 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:13.955480 10046 raft_consensus.cc:504] T 144ff11d2a504f7ab7a230e6822d24ca P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } }
I20240501 07:59:13.955864 10046 leader_election.cc:288] T 144ff11d2a504f7ab7a230e6822d24ca P 3b164cbbc00f4e8eb3254ddd349a3ff9 [CANDIDATE]: Term 1 election: Requested vote from peers 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129:46875), 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149)
I20240501 07:59:13.956048 11901 raft_consensus.cc:504] T 1c32cd00bc344ae1b161d3f9b43b53d3 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } }
I20240501 07:59:13.956264 9189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "144ff11d2a504f7ab7a230e6822d24ca" candidate_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83f20e84af4c40d09a4804b6b26e697f"
I20240501 07:59:13.956383 9189 raft_consensus.cc:3037] T 144ff11d2a504f7ab7a230e6822d24ca P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:13.956388 6311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "144ff11d2a504f7ab7a230e6822d24ca" candidate_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f"
I20240501 07:59:13.956411 11901 leader_election.cc:288] T 1c32cd00bc344ae1b161d3f9b43b53d3 P 8b8b3ad3aa8f4053b2167dd550f2086f [CANDIDATE]: Term 1 election: Requested vote from peers 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132:36773), 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149)
I20240501 07:59:13.956481 6311 raft_consensus.cc:3037] T 144ff11d2a504f7ab7a230e6822d24ca P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:13.956786 9188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c32cd00bc344ae1b161d3f9b43b53d3" candidate_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83f20e84af4c40d09a4804b6b26e697f"
I20240501 07:59:13.956800 6708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c32cd00bc344ae1b161d3f9b43b53d3" candidate_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f9cb72be98941b5a8e504cbe34132b9"
I20240501 07:59:13.956895 9188 raft_consensus.cc:3037] T 1c32cd00bc344ae1b161d3f9b43b53d3 P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:13.956920 6708 raft_consensus.cc:3037] T 1c32cd00bc344ae1b161d3f9b43b53d3 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:13.957417 6311 raft_consensus.cc:2454] T 144ff11d2a504f7ab7a230e6822d24ca P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b164cbbc00f4e8eb3254ddd349a3ff9 in term 1.
I20240501 07:59:13.957417 9189 raft_consensus.cc:2454] T 144ff11d2a504f7ab7a230e6822d24ca P 83f20e84af4c40d09a4804b6b26e697f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b164cbbc00f4e8eb3254ddd349a3ff9 in term 1.
I20240501 07:59:13.957700 6510 leader_election.cc:302] T 144ff11d2a504f7ab7a230e6822d24ca P 3b164cbbc00f4e8eb3254ddd349a3ff9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b164cbbc00f4e8eb3254ddd349a3ff9, 8b8b3ad3aa8f4053b2167dd550f2086f; no voters:
I20240501 07:59:13.957823 9188 raft_consensus.cc:2454] T 1c32cd00bc344ae1b161d3f9b43b53d3 P 83f20e84af4c40d09a4804b6b26e697f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b8b3ad3aa8f4053b2167dd550f2086f in term 1.
I20240501 07:59:13.957865 10046 raft_consensus.cc:2781] T 144ff11d2a504f7ab7a230e6822d24ca P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:13.958027 10046 raft_consensus.cc:686] T 144ff11d2a504f7ab7a230e6822d24ca P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 LEADER]: Becoming Leader. State: Replica: 3b164cbbc00f4e8eb3254ddd349a3ff9, State: Running, Role: LEADER
I20240501 07:59:13.958155 6249 leader_election.cc:302] T 1c32cd00bc344ae1b161d3f9b43b53d3 P 8b8b3ad3aa8f4053b2167dd550f2086f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83f20e84af4c40d09a4804b6b26e697f, 8b8b3ad3aa8f4053b2167dd550f2086f; no voters:
I20240501 07:59:13.958177 10046 consensus_queue.cc:237] T 144ff11d2a504f7ab7a230e6822d24ca P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } }
I20240501 07:59:13.958312 11901 raft_consensus.cc:2781] T 1c32cd00bc344ae1b161d3f9b43b53d3 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:13.958390 11901 raft_consensus.cc:686] T 1c32cd00bc344ae1b161d3f9b43b53d3 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 LEADER]: Becoming Leader. State: Replica: 8b8b3ad3aa8f4053b2167dd550f2086f, State: Running, Role: LEADER
I20240501 07:59:13.958505 11901 consensus_queue.cc:237] T 1c32cd00bc344ae1b161d3f9b43b53d3 P 8b8b3ad3aa8f4053b2167dd550f2086f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } }
I20240501 07:59:13.958551 6708 raft_consensus.cc:2454] T 1c32cd00bc344ae1b161d3f9b43b53d3 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b8b3ad3aa8f4053b2167dd550f2086f in term 1.
I20240501 07:59:13.960660 6172 catalog_manager.cc:5484] T 144ff11d2a504f7ab7a230e6822d24ca P 3b164cbbc00f4e8eb3254ddd349a3ff9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } health_report { overall_health: UNKNOWN } } }
I20240501 07:59:13.960660 6167 catalog_manager.cc:5484] T 1c32cd00bc344ae1b161d3f9b43b53d3 P 8b8b3ad3aa8f4053b2167dd550f2086f reported cstate change: term changed from 0 to 1, leader changed from <none> to 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129). New cstate: current_term: 1 leader_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } health_report { overall_health: UNKNOWN } } }
I20240501 07:59:13.978308 11215 raft_consensus.cc:1050] T 239ee596e62d47169eeb6670dc62d59c P 13d05ebdca8c4f489c597ce550dd773d: attempting to promote NON_VOTER 1f9cb72be98941b5a8e504cbe34132b9 to VOTER
I20240501 07:59:13.978724 11215 consensus_queue.cc:237] T 239ee596e62d47169eeb6670dc62d59c P 13d05ebdca8c4f489c597ce550dd773d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { promote: false } }
I20240501 07:59:13.979969 9056 raft_consensus.cc:1261] T 239ee596e62d47169eeb6670dc62d59c P a505e7cb133c4392bfda8a4a20874ae9 [term 1 FOLLOWER]: Refusing update from remote peer 13d05ebdca8c4f489c597ce550dd773d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 07:59:13.980095 6708 raft_consensus.cc:1261] T 239ee596e62d47169eeb6670dc62d59c P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 LEARNER]: Refusing update from remote peer 13d05ebdca8c4f489c597ce550dd773d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 07:59:13.980116 8793 raft_consensus.cc:1261] T 239ee596e62d47169eeb6670dc62d59c P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Refusing update from remote peer 13d05ebdca8c4f489c597ce550dd773d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 07:59:13.980453 11215 consensus_queue.cc:1035] T 239ee596e62d47169eeb6670dc62d59c P 13d05ebdca8c4f489c597ce550dd773d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 07:59:13.980610 9400 consensus_queue.cc:1035] T 239ee596e62d47169eeb6670dc62d59c P 13d05ebdca8c4f489c597ce550dd773d [LEADER]: Connected to new peer: Peer: permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 07:59:13.980780 12019 consensus_queue.cc:1035] T 239ee596e62d47169eeb6670dc62d59c P 13d05ebdca8c4f489c597ce550dd773d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 07:59:13.982750 11215 raft_consensus.cc:2932] T 239ee596e62d47169eeb6670dc62d59c P 13d05ebdca8c4f489c597ce550dd773d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { promote: false } } }
I20240501 07:59:13.983246 6708 raft_consensus.cc:2932] T 239ee596e62d47169eeb6670dc62d59c P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { promote: false } } }
I20240501 07:59:13.983275 8793 raft_consensus.cc:2932] T 239ee596e62d47169eeb6670dc62d59c P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { promote: false } } }
I20240501 07:59:13.985262 9056 raft_consensus.cc:2932] T 239ee596e62d47169eeb6670dc62d59c P a505e7cb133c4392bfda8a4a20874ae9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { promote: false } } }
I20240501 07:59:13.986310 6172 catalog_manager.cc:5484] T 239ee596e62d47169eeb6670dc62d59c P 74077b61d6f54342bf9736b4a9f2eade reported cstate change: config changed from index 2 to 3, 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "13d05ebdca8c4f489c597ce550dd773d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { promote: false } } }
I20240501 07:59:13.990253 12023 ts_tablet_manager.cc:902] T 6daebcfc41754ec4aca939ca20588854 P 8b8b3ad3aa8f4053b2167dd550f2086f: Initiating tablet copy from peer 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132:36773)
I20240501 07:59:13.990742 8925 consensus_queue.cc:237] T 239ee596e62d47169eeb6670dc62d59c P 13d05ebdca8c4f489c597ce550dd773d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { promote: false } }
I20240501 07:59:13.991266 12023 tablet_copy_client.cc:308] T 6daebcfc41754ec4aca939ca20588854 P 8b8b3ad3aa8f4053b2167dd550f2086f: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:36773
I20240501 07:59:13.991822 6728 tablet_copy_service.cc:140] P 1f9cb72be98941b5a8e504cbe34132b9: Received BeginTabletCopySession request for tablet 6daebcfc41754ec4aca939ca20588854 from peer 8b8b3ad3aa8f4053b2167dd550f2086f ({username='slave'} at 127.0.99.129:52689)
I20240501 07:59:13.991947 6728 tablet_copy_service.cc:161] P 1f9cb72be98941b5a8e504cbe34132b9: Beginning new tablet copy session on tablet 6daebcfc41754ec4aca939ca20588854 from peer 8b8b3ad3aa8f4053b2167dd550f2086f at {username='slave'} at 127.0.99.129:52689: session id = 8b8b3ad3aa8f4053b2167dd550f2086f-6daebcfc41754ec4aca939ca20588854
I20240501 07:59:13.992615 6708 raft_consensus.cc:1261] T 239ee596e62d47169eeb6670dc62d59c P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Refusing update from remote peer 13d05ebdca8c4f489c597ce550dd773d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:59:13.992897 6728 tablet_copy_source_session.cc:205] T 6daebcfc41754ec4aca939ca20588854 P 1f9cb72be98941b5a8e504cbe34132b9: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:13.993037 12019 consensus_queue.cc:1035] T 239ee596e62d47169eeb6670dc62d59c P 13d05ebdca8c4f489c597ce550dd773d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:13.994168 9056 raft_consensus.cc:1261] T 239ee596e62d47169eeb6670dc62d59c P a505e7cb133c4392bfda8a4a20874ae9 [term 1 FOLLOWER]: Refusing update from remote peer 13d05ebdca8c4f489c597ce550dd773d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:59:13.994259 12023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6daebcfc41754ec4aca939ca20588854. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:13.994609 12019 consensus_queue.cc:1035] T 239ee596e62d47169eeb6670dc62d59c P 13d05ebdca8c4f489c597ce550dd773d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:13.994935 10478 raft_consensus.cc:2932] T 239ee596e62d47169eeb6670dc62d59c P 13d05ebdca8c4f489c597ce550dd773d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { promote: false } } }
I20240501 07:59:13.995468 6708 raft_consensus.cc:2932] T 239ee596e62d47169eeb6670dc62d59c P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { promote: false } } }
I20240501 07:59:13.996487 6159 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 239ee596e62d47169eeb6670dc62d59c with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 07:59:13.997332 12023 tablet_copy_client.cc:783] T 6daebcfc41754ec4aca939ca20588854 P 8b8b3ad3aa8f4053b2167dd550f2086f: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:13.997506 12023 tablet_copy_client.cc:647] T 6daebcfc41754ec4aca939ca20588854 P 8b8b3ad3aa8f4053b2167dd550f2086f: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:13.998237 6171 catalog_manager.cc:5484] T 239ee596e62d47169eeb6670dc62d59c P 13d05ebdca8c4f489c597ce550dd773d reported cstate change: config changed from index 3 to 4, VOTER 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133) evicted. New cstate: current_term: 1 leader_uuid: "13d05ebdca8c4f489c597ce550dd773d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 07:59:13.998703 12023 tablet_copy_client.cc:523] T 6daebcfc41754ec4aca939ca20588854 P 8b8b3ad3aa8f4053b2167dd550f2086f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:14.001397 12023 tablet_bootstrap.cc:492] T 6daebcfc41754ec4aca939ca20588854 P 8b8b3ad3aa8f4053b2167dd550f2086f: Bootstrap starting.
I20240501 07:59:14.001521 8757 tablet_service.cc:1514] Processing DeleteTablet for tablet 239ee596e62d47169eeb6670dc62d59c with delete_type TABLET_DATA_TOMBSTONED (TS 74077b61d6f54342bf9736b4a9f2eade not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:43136
I20240501 07:59:14.001935 12007 tablet_replica.cc:330] T 239ee596e62d47169eeb6670dc62d59c P 74077b61d6f54342bf9736b4a9f2eade: stopping tablet replica
I20240501 07:59:14.002061 12007 raft_consensus.cc:2229] T 239ee596e62d47169eeb6670dc62d59c P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:14.002168 12007 raft_consensus.cc:2258] T 239ee596e62d47169eeb6670dc62d59c P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:14.002835 12007 ts_tablet_manager.cc:1875] T 239ee596e62d47169eeb6670dc62d59c P 74077b61d6f54342bf9736b4a9f2eade: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:59:14.004807 12007 ts_tablet_manager.cc:1888] T 239ee596e62d47169eeb6670dc62d59c P 74077b61d6f54342bf9736b4a9f2eade: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 07:59:14.004876 12007 log.cc:1196] T 239ee596e62d47169eeb6670dc62d59c P 74077b61d6f54342bf9736b4a9f2eade: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/239ee596e62d47169eeb6670dc62d59c
I20240501 07:59:14.005367 12023 tablet_bootstrap.cc:492] T 6daebcfc41754ec4aca939ca20588854 P 8b8b3ad3aa8f4053b2167dd550f2086f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:14.005466 6157 catalog_manager.cc:4830] TS 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133:39859): tablet 239ee596e62d47169eeb6670dc62d59c (table rebalancer_extra_table_6 [id=8b10e9a527644d40b1515561fc5df5c2]) successfully deleted
I20240501 07:59:14.005633 12023 tablet_bootstrap.cc:492] T 6daebcfc41754ec4aca939ca20588854 P 8b8b3ad3aa8f4053b2167dd550f2086f: Bootstrap complete.
I20240501 07:59:14.005797 12023 ts_tablet_manager.cc:1367] T 6daebcfc41754ec4aca939ca20588854 P 8b8b3ad3aa8f4053b2167dd550f2086f: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20240501 07:59:14.006242 12023 raft_consensus.cc:348] T 6daebcfc41754ec4aca939ca20588854 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { promote: true } }
I20240501 07:59:14.006352 12023 raft_consensus.cc:729] T 6daebcfc41754ec4aca939ca20588854 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8b8b3ad3aa8f4053b2167dd550f2086f, State: Initialized, Role: LEARNER
I20240501 07:59:14.006457 12023 consensus_queue.cc:260] T 6daebcfc41754ec4aca939ca20588854 P 8b8b3ad3aa8f4053b2167dd550f2086f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { promote: true } }
I20240501 07:59:14.006672 6171 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55950:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 07:59:14.006911 12023 ts_tablet_manager.cc:1398] T 6daebcfc41754ec4aca939ca20588854 P 8b8b3ad3aa8f4053b2167dd550f2086f: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240501 07:59:14.007612 6728 tablet_copy_service.cc:342] P 1f9cb72be98941b5a8e504cbe34132b9: Request end of tablet copy session 8b8b3ad3aa8f4053b2167dd550f2086f-6daebcfc41754ec4aca939ca20588854 received from {username='slave'} at 127.0.99.129:52689
I20240501 07:59:14.007687 6728 tablet_copy_service.cc:433] P 1f9cb72be98941b5a8e504cbe34132b9: ending tablet copy session 8b8b3ad3aa8f4053b2167dd550f2086f-6daebcfc41754ec4aca939ca20588854 on tablet 6daebcfc41754ec4aca939ca20588854 with peer 8b8b3ad3aa8f4053b2167dd550f2086f
I20240501 07:59:14.008045 9056 raft_consensus.cc:2932] T 239ee596e62d47169eeb6670dc62d59c P a505e7cb133c4392bfda8a4a20874ae9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { promote: false } } }
I20240501 07:59:14.013254 8512 raft_consensus.cc:1050] T a756205204d146ce82a2799e30582f57 P 8315d22c56f7439a9e1fc82f153f018c: attempting to promote NON_VOTER a505e7cb133c4392bfda8a4a20874ae9 to VOTER
I20240501 07:59:14.013752 8512 consensus_queue.cc:237] T a756205204d146ce82a2799e30582f57 P 8315d22c56f7439a9e1fc82f153f018c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false } }
I20240501 07:59:14.014720 8905 tablet_service.cc:1467] Processing CreateTablet for tablet 1e06c2e0771541bdb61d1d995c93a543 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=b2142804a6254da283ecbc4af2be1c08]), partition=
I20240501 07:59:14.015043 8905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e06c2e0771541bdb61d1d995c93a543. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:14.015488 9055 raft_consensus.cc:1261] T a756205204d146ce82a2799e30582f57 P a505e7cb133c4392bfda8a4a20874ae9 [term 1 LEARNER]: Refusing update from remote peer 8315d22c56f7439a9e1fc82f153f018c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:59:14.015542 9035 tablet_service.cc:1467] Processing CreateTablet for tablet 1e06c2e0771541bdb61d1d995c93a543 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=b2142804a6254da283ecbc4af2be1c08]), partition=
I20240501 07:59:14.015758 9035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e06c2e0771541bdb61d1d995c93a543. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:14.016180 10111 consensus_queue.cc:1035] T a756205204d146ce82a2799e30582f57 P 8315d22c56f7439a9e1fc82f153f018c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:59:14.016345 6424 tablet_service.cc:1467] Processing CreateTablet for tablet 1e06c2e0771541bdb61d1d995c93a543 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=b2142804a6254da283ecbc4af2be1c08]), partition=
I20240501 07:59:14.016557 6424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e06c2e0771541bdb61d1d995c93a543. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:14.018769 12035 tablet_bootstrap.cc:492] T 1e06c2e0771541bdb61d1d995c93a543 P 8315d22c56f7439a9e1fc82f153f018c: Bootstrap starting.
I20240501 07:59:14.019404 6577 raft_consensus.cc:1261] T a756205204d146ce82a2799e30582f57 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Refusing update from remote peer 8315d22c56f7439a9e1fc82f153f018c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:59:14.019814 12035 tablet_bootstrap.cc:654] T 1e06c2e0771541bdb61d1d995c93a543 P 8315d22c56f7439a9e1fc82f153f018c: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:14.019924 11735 consensus_queue.cc:1035] T a756205204d146ce82a2799e30582f57 P 8315d22c56f7439a9e1fc82f153f018c [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:59:14.021126 6708 raft_consensus.cc:1261] T a756205204d146ce82a2799e30582f57 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Refusing update from remote peer 8315d22c56f7439a9e1fc82f153f018c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:59:14.024945 10111 consensus_queue.cc:1035] T a756205204d146ce82a2799e30582f57 P 8315d22c56f7439a9e1fc82f153f018c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:59:14.028424 8512 raft_consensus.cc:2932] T a756205204d146ce82a2799e30582f57 P 8315d22c56f7439a9e1fc82f153f018c [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false } } }
I20240501 07:59:14.028816 6708 raft_consensus.cc:2932] T a756205204d146ce82a2799e30582f57 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false } } }
I20240501 07:59:14.029520 11987 tablet_bootstrap.cc:492] T 1e06c2e0771541bdb61d1d995c93a543 P 13d05ebdca8c4f489c597ce550dd773d: Bootstrap starting.
I20240501 07:59:14.030642 11987 tablet_bootstrap.cc:654] T 1e06c2e0771541bdb61d1d995c93a543 P 13d05ebdca8c4f489c597ce550dd773d: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:14.031213 6577 raft_consensus.cc:2932] T a756205204d146ce82a2799e30582f57 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false } } }
I20240501 07:59:14.032409 11987 tablet_bootstrap.cc:492] T 1e06c2e0771541bdb61d1d995c93a543 P 13d05ebdca8c4f489c597ce550dd773d: No bootstrap required, opened a new log
I20240501 07:59:14.032503 11987 ts_tablet_manager.cc:1367] T 1e06c2e0771541bdb61d1d995c93a543 P 13d05ebdca8c4f489c597ce550dd773d: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20240501 07:59:14.033586 11987 raft_consensus.cc:348] T 1e06c2e0771541bdb61d1d995c93a543 P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } }
I20240501 07:59:14.033792 11987 raft_consensus.cc:374] T 1e06c2e0771541bdb61d1d995c93a543 P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:14.033878 11987 raft_consensus.cc:729] T 1e06c2e0771541bdb61d1d995c93a543 P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13d05ebdca8c4f489c597ce550dd773d, State: Initialized, Role: FOLLOWER
I20240501 07:59:14.033968 6171 catalog_manager.cc:5484] T a756205204d146ce82a2799e30582f57 P 1f9cb72be98941b5a8e504cbe34132b9 reported cstate change: config changed from index 4 to 5, a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8315d22c56f7439a9e1fc82f153f018c" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false } } }
I20240501 07:59:14.034107 11987 consensus_queue.cc:260] T 1e06c2e0771541bdb61d1d995c93a543 P 13d05ebdca8c4f489c597ce550dd773d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } }
I20240501 07:59:14.034214 12037 tablet_bootstrap.cc:492] T 1e06c2e0771541bdb61d1d995c93a543 P a505e7cb133c4392bfda8a4a20874ae9: Bootstrap starting.
I20240501 07:59:14.034484 11987 ts_tablet_manager.cc:1398] T 1e06c2e0771541bdb61d1d995c93a543 P 13d05ebdca8c4f489c597ce550dd773d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:59:14.035518 12023 ts_tablet_manager.cc:902] T 54b3efabad7f4b608cc9d931546b5743 P 8b8b3ad3aa8f4053b2167dd550f2086f: Initiating tablet copy from peer 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130:32941)
I20240501 07:59:14.036005 12023 tablet_copy_client.cc:308] T 54b3efabad7f4b608cc9d931546b5743 P 8b8b3ad3aa8f4053b2167dd550f2086f: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:32941
I20240501 07:59:14.036473 6466 tablet_copy_service.cc:140] P 8315d22c56f7439a9e1fc82f153f018c: Received BeginTabletCopySession request for tablet 54b3efabad7f4b608cc9d931546b5743 from peer 8b8b3ad3aa8f4053b2167dd550f2086f ({username='slave'} at 127.0.99.129:45145)
I20240501 07:59:14.036564 6466 tablet_copy_service.cc:161] P 8315d22c56f7439a9e1fc82f153f018c: Beginning new tablet copy session on tablet 54b3efabad7f4b608cc9d931546b5743 from peer 8b8b3ad3aa8f4053b2167dd550f2086f at {username='slave'} at 127.0.99.129:45145: session id = 8b8b3ad3aa8f4053b2167dd550f2086f-54b3efabad7f4b608cc9d931546b5743
I20240501 07:59:14.037158 9055 raft_consensus.cc:2932] T a756205204d146ce82a2799e30582f57 P a505e7cb133c4392bfda8a4a20874ae9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false } } }
I20240501 07:59:14.037479 12037 tablet_bootstrap.cc:654] T 1e06c2e0771541bdb61d1d995c93a543 P a505e7cb133c4392bfda8a4a20874ae9: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:14.037569 6466 tablet_copy_source_session.cc:205] T 54b3efabad7f4b608cc9d931546b5743 P 8315d22c56f7439a9e1fc82f153f018c: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:14.041016 10324 raft_consensus.cc:1050] T 8663a511521644009fc360ad92c093a1 P 1f9cb72be98941b5a8e504cbe34132b9: attempting to promote NON_VOTER 13d05ebdca8c4f489c597ce550dd773d to VOTER
I20240501 07:59:14.041201 12023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54b3efabad7f4b608cc9d931546b5743. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:14.041445 10324 consensus_queue.cc:237] T 8663a511521644009fc360ad92c093a1 P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false } }
I20240501 07:59:14.043567 8925 raft_consensus.cc:1261] T 8663a511521644009fc360ad92c093a1 P 13d05ebdca8c4f489c597ce550dd773d [term 1 LEARNER]: Refusing update from remote peer 1f9cb72be98941b5a8e504cbe34132b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:59:14.043828 6577 raft_consensus.cc:1261] T 8663a511521644009fc360ad92c093a1 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Refusing update from remote peer 1f9cb72be98941b5a8e504cbe34132b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:59:14.044250 11402 consensus_queue.cc:1035] T 8663a511521644009fc360ad92c093a1 P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:59:14.044641 11719 consensus_queue.cc:1035] T 8663a511521644009fc360ad92c093a1 P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:59:14.045344 6444 consensus_queue.cc:237] T a756205204d146ce82a2799e30582f57 P 8315d22c56f7439a9e1fc82f153f018c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false } }
I20240501 07:59:14.045532 6445 raft_consensus.cc:1261] T 8663a511521644009fc360ad92c093a1 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Refusing update from remote peer 1f9cb72be98941b5a8e504cbe34132b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:59:14.046938 9056 raft_consensus.cc:1261] T a756205204d146ce82a2799e30582f57 P a505e7cb133c4392bfda8a4a20874ae9 [term 1 FOLLOWER]: Refusing update from remote peer 8315d22c56f7439a9e1fc82f153f018c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 07:59:14.047461 12011 consensus_queue.cc:1035] T a756205204d146ce82a2799e30582f57 P 8315d22c56f7439a9e1fc82f153f018c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 07:59:14.048636 11719 consensus_queue.cc:1035] T 8663a511521644009fc360ad92c093a1 P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:59:14.049481 10478 raft_consensus.cc:482] T 1e06c2e0771541bdb61d1d995c93a543 P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:14.049587 10478 raft_consensus.cc:504] T 1e06c2e0771541bdb61d1d995c93a543 P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } }
I20240501 07:59:14.050041 10478 leader_election.cc:288] T 1e06c2e0771541bdb61d1d995c93a543 P 13d05ebdca8c4f489c597ce550dd773d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607), 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130:32941)
I20240501 07:59:14.050709 6445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e06c2e0771541bdb61d1d995c93a543" candidate_uuid: "13d05ebdca8c4f489c597ce550dd773d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8315d22c56f7439a9e1fc82f153f018c" is_pre_election: true
W20240501 07:59:14.051095 8860 leader_election.cc:341] T 1e06c2e0771541bdb61d1d995c93a543 P 13d05ebdca8c4f489c597ce550dd773d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130:32941): Illegal state: must be running to vote when last-logged opid is not known
I20240501 07:59:14.050034 6708 raft_consensus.cc:1261] T a756205204d146ce82a2799e30582f57 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Refusing update from remote peer 8315d22c56f7439a9e1fc82f153f018c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 07:59:14.052178 9056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e06c2e0771541bdb61d1d995c93a543" candidate_uuid: "13d05ebdca8c4f489c597ce550dd773d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a505e7cb133c4392bfda8a4a20874ae9" is_pre_election: true
W20240501 07:59:14.052584 8859 leader_election.cc:341] T 1e06c2e0771541bdb61d1d995c93a543 P 13d05ebdca8c4f489c597ce550dd773d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Illegal state: must be running to vote when last-logged opid is not known
I20240501 07:59:14.052677 8859 leader_election.cc:302] T 1e06c2e0771541bdb61d1d995c93a543 P 13d05ebdca8c4f489c597ce550dd773d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 13d05ebdca8c4f489c597ce550dd773d; no voters: 8315d22c56f7439a9e1fc82f153f018c, a505e7cb133c4392bfda8a4a20874ae9
I20240501 07:59:14.052824 10478 raft_consensus.cc:2726] T 1e06c2e0771541bdb61d1d995c93a543 P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 07:59:14.052626 12048 raft_consensus.cc:2932] T 8663a511521644009fc360ad92c093a1 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false } } }
I20240501 07:59:14.052933 12011 consensus_queue.cc:1035] T a756205204d146ce82a2799e30582f57 P 8315d22c56f7439a9e1fc82f153f018c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 07:59:14.053931 8924 raft_consensus.cc:2932] T 8663a511521644009fc360ad92c093a1 P 13d05ebdca8c4f489c597ce550dd773d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false } } }
I20240501 07:59:14.055418 6577 raft_consensus.cc:2932] T 8663a511521644009fc360ad92c093a1 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false } } }
I20240501 07:59:14.056147 6172 catalog_manager.cc:5484] T 8663a511521644009fc360ad92c093a1 P 1f9cb72be98941b5a8e504cbe34132b9 reported cstate change: config changed from index 4 to 5, 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1f9cb72be98941b5a8e504cbe34132b9" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 07:59:14.058385 12037 tablet_bootstrap.cc:492] T 1e06c2e0771541bdb61d1d995c93a543 P a505e7cb133c4392bfda8a4a20874ae9: No bootstrap required, opened a new log
I20240501 07:59:14.058502 12037 ts_tablet_manager.cc:1367] T 1e06c2e0771541bdb61d1d995c93a543 P a505e7cb133c4392bfda8a4a20874ae9: Time spent bootstrapping tablet: real 0.024s user 0.002s sys 0.000s
I20240501 07:59:14.059051 12037 raft_consensus.cc:348] T 1e06c2e0771541bdb61d1d995c93a543 P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } }
I20240501 07:59:14.059173 12037 raft_consensus.cc:374] T 1e06c2e0771541bdb61d1d995c93a543 P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:14.059206 12037 raft_consensus.cc:729] T 1e06c2e0771541bdb61d1d995c93a543 P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a505e7cb133c4392bfda8a4a20874ae9, State: Initialized, Role: FOLLOWER
I20240501 07:59:14.059330 12037 consensus_queue.cc:260] T 1e06c2e0771541bdb61d1d995c93a543 P a505e7cb133c4392bfda8a4a20874ae9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } }
I20240501 07:59:14.060526 6707 consensus_queue.cc:237] T 8663a511521644009fc360ad92c093a1 P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false } }
I20240501 07:59:14.061056 12037 ts_tablet_manager.cc:1398] T 1e06c2e0771541bdb61d1d995c93a543 P a505e7cb133c4392bfda8a4a20874ae9: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 07:59:14.062847 8924 raft_consensus.cc:1261] T 8663a511521644009fc360ad92c093a1 P 13d05ebdca8c4f489c597ce550dd773d [term 1 FOLLOWER]: Refusing update from remote peer 1f9cb72be98941b5a8e504cbe34132b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 07:59:14.063396 11402 consensus_queue.cc:1035] T 8663a511521644009fc360ad92c093a1 P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 07:59:14.064154 6577 raft_consensus.cc:1261] T 8663a511521644009fc360ad92c093a1 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Refusing update from remote peer 1f9cb72be98941b5a8e504cbe34132b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 07:59:14.065011 10111 raft_consensus.cc:2932] T a756205204d146ce82a2799e30582f57 P 8315d22c56f7439a9e1fc82f153f018c [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false } } }
I20240501 07:59:14.065232 11719 consensus_queue.cc:1035] T 8663a511521644009fc360ad92c093a1 P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 07:59:14.066915 11719 raft_consensus.cc:2932] T 8663a511521644009fc360ad92c093a1 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false } } }
I20240501 07:59:14.068385 6167 catalog_manager.cc:5484] T a756205204d146ce82a2799e30582f57 P 8315d22c56f7439a9e1fc82f153f018c reported cstate change: config changed from index 5 to 6, VOTER 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "8315d22c56f7439a9e1fc82f153f018c" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 07:59:14.068590 6159 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8663a511521644009fc360ad92c093a1 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 07:59:14.069351 6159 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a756205204d146ce82a2799e30582f57 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 07:59:14.068873 9055 raft_consensus.cc:2932] T a756205204d146ce82a2799e30582f57 P a505e7cb133c4392bfda8a4a20874ae9 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false } } }
I20240501 07:59:14.070231 6708 raft_consensus.cc:2932] T a756205204d146ce82a2799e30582f57 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false } } }
I20240501 07:59:14.071209 6171 catalog_manager.cc:5484] T 8663a511521644009fc360ad92c093a1 P 1f9cb72be98941b5a8e504cbe34132b9 reported cstate change: config changed from index 5 to 6, VOTER 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "1f9cb72be98941b5a8e504cbe34132b9" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 07:59:14.073144 12023 tablet_copy_client.cc:783] T 54b3efabad7f4b608cc9d931546b5743 P 8b8b3ad3aa8f4053b2167dd550f2086f: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:14.073374 12023 tablet_copy_client.cc:647] T 54b3efabad7f4b608cc9d931546b5743 P 8b8b3ad3aa8f4053b2167dd550f2086f: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:14.073760 6424 tablet_service.cc:1514] Processing DeleteTablet for tablet 8663a511521644009fc360ad92c093a1 with delete_type TABLET_DATA_TOMBSTONED (TS 8315d22c56f7439a9e1fc82f153f018c not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:45034
I20240501 07:59:14.075121 6577 raft_consensus.cc:2932] T 8663a511521644009fc360ad92c093a1 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false } } }
I20240501 07:59:14.075218 8924 raft_consensus.cc:2932] T 8663a511521644009fc360ad92c093a1 P 13d05ebdca8c4f489c597ce550dd773d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false } } }
I20240501 07:59:14.075790 6556 tablet_service.cc:1514] Processing DeleteTablet for tablet a756205204d146ce82a2799e30582f57 with delete_type TABLET_DATA_TOMBSTONED (TS 3b164cbbc00f4e8eb3254ddd349a3ff9 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:53874
I20240501 07:59:14.075982 11758 tablet_replica.cc:330] T a756205204d146ce82a2799e30582f57 P 3b164cbbc00f4e8eb3254ddd349a3ff9: stopping tablet replica
I20240501 07:59:14.076113 11758 raft_consensus.cc:2229] T a756205204d146ce82a2799e30582f57 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:14.076294 11758 raft_consensus.cc:2258] T a756205204d146ce82a2799e30582f57 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:14.077136 11758 ts_tablet_manager.cc:1875] T a756205204d146ce82a2799e30582f57 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:59:14.079499 11758 ts_tablet_manager.cc:1888] T a756205204d146ce82a2799e30582f57 P 3b164cbbc00f4e8eb3254ddd349a3ff9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 07:59:14.079571 11758 log.cc:1196] T a756205204d146ce82a2799e30582f57 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/a756205204d146ce82a2799e30582f57
I20240501 07:59:14.080106 6160 catalog_manager.cc:4830] TS 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131:34471): tablet a756205204d146ce82a2799e30582f57 (table rebalance_test_table_9 [id=c4f46595186f4427aa00d6752aa5dc9a]) successfully deleted
I20240501 07:59:14.080879 12035 tablet_bootstrap.cc:492] T 1e06c2e0771541bdb61d1d995c93a543 P 8315d22c56f7439a9e1fc82f153f018c: No bootstrap required, opened a new log
I20240501 07:59:14.081030 12035 ts_tablet_manager.cc:1367] T 1e06c2e0771541bdb61d1d995c93a543 P 8315d22c56f7439a9e1fc82f153f018c: Time spent bootstrapping tablet: real 0.062s user 0.002s sys 0.000s
I20240501 07:59:14.081519 12035 raft_consensus.cc:348] T 1e06c2e0771541bdb61d1d995c93a543 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } }
I20240501 07:59:14.081645 12035 raft_consensus.cc:374] T 1e06c2e0771541bdb61d1d995c93a543 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:14.081682 12035 raft_consensus.cc:729] T 1e06c2e0771541bdb61d1d995c93a543 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8315d22c56f7439a9e1fc82f153f018c, State: Initialized, Role: FOLLOWER
I20240501 07:59:14.081832 12035 consensus_queue.cc:260] T 1e06c2e0771541bdb61d1d995c93a543 P 8315d22c56f7439a9e1fc82f153f018c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } }
I20240501 07:59:14.082192 12035 ts_tablet_manager.cc:1398] T 1e06c2e0771541bdb61d1d995c93a543 P 8315d22c56f7439a9e1fc82f153f018c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:14.085145 12023 tablet_copy_client.cc:523] T 54b3efabad7f4b608cc9d931546b5743 P 8b8b3ad3aa8f4053b2167dd550f2086f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:14.086455 12023 tablet_bootstrap.cc:492] T 54b3efabad7f4b608cc9d931546b5743 P 8b8b3ad3aa8f4053b2167dd550f2086f: Bootstrap starting.
I20240501 07:59:14.087110 12057 tablet_replica.cc:330] T 8663a511521644009fc360ad92c093a1 P 8315d22c56f7439a9e1fc82f153f018c: stopping tablet replica
I20240501 07:59:14.087242 12057 raft_consensus.cc:2229] T 8663a511521644009fc360ad92c093a1 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:14.087388 12057 pending_rounds.cc:62] T 8663a511521644009fc360ad92c093a1 P 8315d22c56f7439a9e1fc82f153f018c: Trying to abort 1 pending ops.
I20240501 07:59:14.087455 12057 pending_rounds.cc:68] T 8663a511521644009fc360ad92c093a1 P 8315d22c56f7439a9e1fc82f153f018c: Aborting op as it isn't in flight: id { term: 1 index: 5 } timestamp: 7022798250152677376 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "8663a511521644009fc360ad92c093a1" old_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: true } } } new_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false } } } }
I20240501 07:59:14.087639 12057 raft_consensus.cc:2894] T 8663a511521644009fc360ad92c093a1 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Aborting config change with OpId 1.5: Aborted: Op aborted
I20240501 07:59:14.087852 12058 ts_tablet_manager.cc:902] T 2e540f389b2a4aef93a020f549d11b67 P 13d05ebdca8c4f489c597ce550dd773d: Initiating tablet copy from peer 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129:46875)
I20240501 07:59:14.088485 12058 tablet_copy_client.cc:308] T 2e540f389b2a4aef93a020f549d11b67 P 13d05ebdca8c4f489c597ce550dd773d: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:46875
I20240501 07:59:14.088944 12057 raft_consensus.cc:2866] T 8663a511521644009fc360ad92c093a1 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240501 07:59:14.088590 12063 ts_tablet_manager.cc:902] T 542c9ac08f7d4bec9b15a309c2e21364 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Initiating tablet copy from peer 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134:40141)
I20240501 07:59:14.089602 6335 tablet_copy_service.cc:140] P 8b8b3ad3aa8f4053b2167dd550f2086f: Received BeginTabletCopySession request for tablet 2e540f389b2a4aef93a020f549d11b67 from peer 13d05ebdca8c4f489c597ce550dd773d ({username='slave'} at 127.0.99.134:55617)
I20240501 07:59:14.089704 6335 tablet_copy_service.cc:161] P 8b8b3ad3aa8f4053b2167dd550f2086f: Beginning new tablet copy session on tablet 2e540f389b2a4aef93a020f549d11b67 from peer 13d05ebdca8c4f489c597ce550dd773d at {username='slave'} at 127.0.99.134:55617: session id = 13d05ebdca8c4f489c597ce550dd773d-2e540f389b2a4aef93a020f549d11b67
I20240501 07:59:14.089722 12063 tablet_copy_client.cc:308] T 542c9ac08f7d4bec9b15a309c2e21364 P 3b164cbbc00f4e8eb3254ddd349a3ff9: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.134:40141
I20240501 07:59:14.090680 6335 tablet_copy_source_session.cc:205] T 2e540f389b2a4aef93a020f549d11b67 P 8b8b3ad3aa8f4053b2167dd550f2086f: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:14.091042 8945 tablet_copy_service.cc:140] P 13d05ebdca8c4f489c597ce550dd773d: Received BeginTabletCopySession request for tablet 542c9ac08f7d4bec9b15a309c2e21364 from peer 3b164cbbc00f4e8eb3254ddd349a3ff9 ({username='slave'} at 127.0.99.131:60487)
I20240501 07:59:14.091135 8945 tablet_copy_service.cc:161] P 13d05ebdca8c4f489c597ce550dd773d: Beginning new tablet copy session on tablet 542c9ac08f7d4bec9b15a309c2e21364 from peer 3b164cbbc00f4e8eb3254ddd349a3ff9 at {username='slave'} at 127.0.99.131:60487: session id = 3b164cbbc00f4e8eb3254ddd349a3ff9-542c9ac08f7d4bec9b15a309c2e21364
I20240501 07:59:14.091452 12023 tablet_bootstrap.cc:492] T 54b3efabad7f4b608cc9d931546b5743 P 8b8b3ad3aa8f4053b2167dd550f2086f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:14.091692 12023 tablet_bootstrap.cc:492] T 54b3efabad7f4b608cc9d931546b5743 P 8b8b3ad3aa8f4053b2167dd550f2086f: Bootstrap complete.
I20240501 07:59:14.091853 12023 ts_tablet_manager.cc:1367] T 54b3efabad7f4b608cc9d931546b5743 P 8b8b3ad3aa8f4053b2167dd550f2086f: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240501 07:59:14.092141 8945 tablet_copy_source_session.cc:205] T 542c9ac08f7d4bec9b15a309c2e21364 P 13d05ebdca8c4f489c597ce550dd773d: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:14.092391 12023 raft_consensus.cc:348] T 54b3efabad7f4b608cc9d931546b5743 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { replace: true } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { promote: true } }
I20240501 07:59:14.092496 12023 raft_consensus.cc:729] T 54b3efabad7f4b608cc9d931546b5743 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8b8b3ad3aa8f4053b2167dd550f2086f, State: Initialized, Role: LEARNER
I20240501 07:59:14.092607 12023 consensus_queue.cc:260] T 54b3efabad7f4b608cc9d931546b5743 P 8b8b3ad3aa8f4053b2167dd550f2086f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { replace: true } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { promote: true } }
I20240501 07:59:14.092909 12023 ts_tablet_manager.cc:1398] T 54b3efabad7f4b608cc9d931546b5743 P 8b8b3ad3aa8f4053b2167dd550f2086f: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240501 07:59:14.093241 12057 raft_consensus.cc:2258] T 8663a511521644009fc360ad92c093a1 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:14.093442 12063 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 542c9ac08f7d4bec9b15a309c2e21364. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:14.093713 12058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e540f389b2a4aef93a020f549d11b67. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:14.094246 6466 tablet_copy_service.cc:342] P 8315d22c56f7439a9e1fc82f153f018c: Request end of tablet copy session 8b8b3ad3aa8f4053b2167dd550f2086f-54b3efabad7f4b608cc9d931546b5743 received from {username='slave'} at 127.0.99.129:45145
I20240501 07:59:14.094323 6466 tablet_copy_service.cc:433] P 8315d22c56f7439a9e1fc82f153f018c: ending tablet copy session 8b8b3ad3aa8f4053b2167dd550f2086f-54b3efabad7f4b608cc9d931546b5743 on tablet 54b3efabad7f4b608cc9d931546b5743 with peer 8b8b3ad3aa8f4053b2167dd550f2086f
I20240501 07:59:14.097697 12063 tablet_copy_client.cc:783] T 542c9ac08f7d4bec9b15a309c2e21364 P 3b164cbbc00f4e8eb3254ddd349a3ff9: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:14.097868 12063 tablet_copy_client.cc:647] T 542c9ac08f7d4bec9b15a309c2e21364 P 3b164cbbc00f4e8eb3254ddd349a3ff9: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:14.097846 12057 ts_tablet_manager.cc:1875] T 8663a511521644009fc360ad92c093a1 P 8315d22c56f7439a9e1fc82f153f018c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:59:14.100049 12057 ts_tablet_manager.cc:1888] T 8663a511521644009fc360ad92c093a1 P 8315d22c56f7439a9e1fc82f153f018c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 07:59:14.100124 12057 log.cc:1196] T 8663a511521644009fc360ad92c093a1 P 8315d22c56f7439a9e1fc82f153f018c: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/8663a511521644009fc360ad92c093a1
I20240501 07:59:14.100618 6159 catalog_manager.cc:4830] TS 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130:32941): tablet 8663a511521644009fc360ad92c093a1 (table rebalance_test_table_3 [id=eb6765f7643442ff82fde7f790b2aaed]) successfully deleted
I20240501 07:59:14.101183 12063 tablet_copy_client.cc:523] T 542c9ac08f7d4bec9b15a309c2e21364 P 3b164cbbc00f4e8eb3254ddd349a3ff9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:14.102380 12063 tablet_bootstrap.cc:492] T 542c9ac08f7d4bec9b15a309c2e21364 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Bootstrap starting.
I20240501 07:59:14.102836 12058 tablet_copy_client.cc:783] T 2e540f389b2a4aef93a020f549d11b67 P 13d05ebdca8c4f489c597ce550dd773d: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:14.105136 12058 tablet_copy_client.cc:647] T 2e540f389b2a4aef93a020f549d11b67 P 13d05ebdca8c4f489c597ce550dd773d: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:14.106426 12058 tablet_copy_client.cc:523] T 2e540f389b2a4aef93a020f549d11b67 P 13d05ebdca8c4f489c597ce550dd773d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:14.106633 12063 tablet_bootstrap.cc:492] T 542c9ac08f7d4bec9b15a309c2e21364 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:14.106865 12063 tablet_bootstrap.cc:492] T 542c9ac08f7d4bec9b15a309c2e21364 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Bootstrap complete.
I20240501 07:59:14.107031 12063 ts_tablet_manager.cc:1367] T 542c9ac08f7d4bec9b15a309c2e21364 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240501 07:59:14.107425 12063 raft_consensus.cc:348] T 542c9ac08f7d4bec9b15a309c2e21364 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { replace: true } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: true } }
I20240501 07:59:14.107532 12063 raft_consensus.cc:729] T 542c9ac08f7d4bec9b15a309c2e21364 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3b164cbbc00f4e8eb3254ddd349a3ff9, State: Initialized, Role: LEARNER
I20240501 07:59:14.107694 12063 consensus_queue.cc:260] T 542c9ac08f7d4bec9b15a309c2e21364 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { replace: true } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: true } }
I20240501 07:59:14.107757 12058 tablet_bootstrap.cc:492] T 2e540f389b2a4aef93a020f549d11b67 P 13d05ebdca8c4f489c597ce550dd773d: Bootstrap starting.
I20240501 07:59:14.107983 12063 ts_tablet_manager.cc:1398] T 542c9ac08f7d4bec9b15a309c2e21364 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240501 07:59:14.108824 8945 tablet_copy_service.cc:342] P 13d05ebdca8c4f489c597ce550dd773d: Request end of tablet copy session 3b164cbbc00f4e8eb3254ddd349a3ff9-542c9ac08f7d4bec9b15a309c2e21364 received from {username='slave'} at 127.0.99.131:60487
I20240501 07:59:14.108912 8945 tablet_copy_service.cc:433] P 13d05ebdca8c4f489c597ce550dd773d: ending tablet copy session 3b164cbbc00f4e8eb3254ddd349a3ff9-542c9ac08f7d4bec9b15a309c2e21364 on tablet 542c9ac08f7d4bec9b15a309c2e21364 with peer 3b164cbbc00f4e8eb3254ddd349a3ff9
I20240501 07:59:14.116120 6311 raft_consensus.cc:1261] T 6e91683ff22243c1b3877733d74e2886 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 2 FOLLOWER]: Refusing update from remote peer 74077b61d6f54342bf9736b4a9f2eade: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 07:59:14.116550 11723 consensus_queue.cc:1035] T 6e91683ff22243c1b3877733d74e2886 P 74077b61d6f54342bf9736b4a9f2eade [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 07:59:14.144795 12058 tablet_bootstrap.cc:492] T 2e540f389b2a4aef93a020f549d11b67 P 13d05ebdca8c4f489c597ce550dd773d: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:14.145180 12058 tablet_bootstrap.cc:492] T 2e540f389b2a4aef93a020f549d11b67 P 13d05ebdca8c4f489c597ce550dd773d: Bootstrap complete.
I20240501 07:59:14.145457 12058 ts_tablet_manager.cc:1367] T 2e540f389b2a4aef93a020f549d11b67 P 13d05ebdca8c4f489c597ce550dd773d: Time spent bootstrapping tablet: real 0.038s user 0.024s sys 0.004s
I20240501 07:59:14.146055 12058 raft_consensus.cc:348] T 2e540f389b2a4aef93a020f549d11b67 P 13d05ebdca8c4f489c597ce550dd773d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: true } }
I20240501 07:59:14.146265 12058 raft_consensus.cc:729] T 2e540f389b2a4aef93a020f549d11b67 P 13d05ebdca8c4f489c597ce550dd773d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 13d05ebdca8c4f489c597ce550dd773d, State: Initialized, Role: LEARNER
I20240501 07:59:14.146430 12058 consensus_queue.cc:260] T 2e540f389b2a4aef93a020f549d11b67 P 13d05ebdca8c4f489c597ce550dd773d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: true } }
I20240501 07:59:14.146817 12058 ts_tablet_manager.cc:1398] T 2e540f389b2a4aef93a020f549d11b67 P 13d05ebdca8c4f489c597ce550dd773d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:14.147615 6335 tablet_copy_service.cc:342] P 8b8b3ad3aa8f4053b2167dd550f2086f: Request end of tablet copy session 13d05ebdca8c4f489c597ce550dd773d-2e540f389b2a4aef93a020f549d11b67 received from {username='slave'} at 127.0.99.134:55617
I20240501 07:59:14.147703 6335 tablet_copy_service.cc:433] P 8b8b3ad3aa8f4053b2167dd550f2086f: ending tablet copy session 13d05ebdca8c4f489c597ce550dd773d-2e540f389b2a4aef93a020f549d11b67 on tablet 2e540f389b2a4aef93a020f549d11b67 with peer 13d05ebdca8c4f489c597ce550dd773d
I20240501 07:59:14.176100 10046 raft_consensus.cc:1050] T d6c28cd91f624673b8e428f471a1dcce P 3b164cbbc00f4e8eb3254ddd349a3ff9: attempting to promote NON_VOTER a505e7cb133c4392bfda8a4a20874ae9 to VOTER
I20240501 07:59:14.176631 10046 consensus_queue.cc:237] T d6c28cd91f624673b8e428f471a1dcce P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { replace: true } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false } }
I20240501 07:59:14.178309 6444 raft_consensus.cc:1261] T d6c28cd91f624673b8e428f471a1dcce P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Refusing update from remote peer 3b164cbbc00f4e8eb3254ddd349a3ff9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:59:14.178705 6311 raft_consensus.cc:1261] T d6c28cd91f624673b8e428f471a1dcce P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Refusing update from remote peer 3b164cbbc00f4e8eb3254ddd349a3ff9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:59:14.178774 12119 consensus_queue.cc:1035] T d6c28cd91f624673b8e428f471a1dcce P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:59:14.178984 6577 raft_consensus.cc:1261] T 6e91683ff22243c1b3877733d74e2886 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 2 FOLLOWER]: Refusing update from remote peer 74077b61d6f54342bf9736b4a9f2eade: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 07:59:14.179198 12119 consensus_queue.cc:1035] T d6c28cd91f624673b8e428f471a1dcce P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:59:14.179387 9055 raft_consensus.cc:1261] T d6c28cd91f624673b8e428f471a1dcce P a505e7cb133c4392bfda8a4a20874ae9 [term 1 LEARNER]: Refusing update from remote peer 3b164cbbc00f4e8eb3254ddd349a3ff9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:59:14.179466 11723 consensus_queue.cc:1035] T 6e91683ff22243c1b3877733d74e2886 P 74077b61d6f54342bf9736b4a9f2eade [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 07:59:14.179855 11119 consensus_queue.cc:1035] T d6c28cd91f624673b8e428f471a1dcce P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:59:14.182103 6708 raft_consensus.cc:1261] T 6e91683ff22243c1b3877733d74e2886 P 1f9cb72be98941b5a8e504cbe34132b9 [term 2 FOLLOWER]: Refusing update from remote peer 74077b61d6f54342bf9736b4a9f2eade: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 07:59:14.182857 12123 consensus_queue.cc:1035] T 6e91683ff22243c1b3877733d74e2886 P 74077b61d6f54342bf9736b4a9f2eade [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 07:59:14.184187 10046 raft_consensus.cc:2932] T d6c28cd91f624673b8e428f471a1dcce P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { replace: true } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false } } }
I20240501 07:59:14.186069 9055 raft_consensus.cc:2932] T d6c28cd91f624673b8e428f471a1dcce P a505e7cb133c4392bfda8a4a20874ae9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { replace: true } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false } } }
I20240501 07:59:14.186623 6313 raft_consensus.cc:2932] T d6c28cd91f624673b8e428f471a1dcce P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { replace: true } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false } } }
I20240501 07:59:14.188866 6172 catalog_manager.cc:5484] T d6c28cd91f624673b8e428f471a1dcce P a505e7cb133c4392bfda8a4a20874ae9 reported cstate change: config changed from index 4 to 5, a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { replace: true } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false } } }
I20240501 07:59:14.189404 6444 raft_consensus.cc:2932] T d6c28cd91f624673b8e428f471a1dcce P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { replace: true } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false } } }
I20240501 07:59:14.195147 6577 consensus_queue.cc:237] T d6c28cd91f624673b8e428f471a1dcce P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false } }
I20240501 07:59:14.196238 8924 raft_consensus.cc:1261] T 72aa4dc7664b48b7a73d34feba7d44d3 P 13d05ebdca8c4f489c597ce550dd773d [term 2 FOLLOWER]: Refusing update from remote peer a505e7cb133c4392bfda8a4a20874ae9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 07:59:14.196316 9055 raft_consensus.cc:1261] T d6c28cd91f624673b8e428f471a1dcce P a505e7cb133c4392bfda8a4a20874ae9 [term 1 FOLLOWER]: Refusing update from remote peer 3b164cbbc00f4e8eb3254ddd349a3ff9: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 07:59:14.196410 6444 raft_consensus.cc:1261] T d6c28cd91f624673b8e428f471a1dcce P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Refusing update from remote peer 3b164cbbc00f4e8eb3254ddd349a3ff9: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 07:59:14.196828 10046 consensus_queue.cc:1035] T d6c28cd91f624673b8e428f471a1dcce P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 07:59:14.196847 11985 consensus_queue.cc:1035] T 72aa4dc7664b48b7a73d34feba7d44d3 P a505e7cb133c4392bfda8a4a20874ae9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:14.197204 10046 consensus_queue.cc:1035] T d6c28cd91f624673b8e428f471a1dcce P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 07:59:14.199368 10046 raft_consensus.cc:2932] T d6c28cd91f624673b8e428f471a1dcce P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false } } }
I20240501 07:59:14.200217 6444 raft_consensus.cc:2932] T d6c28cd91f624673b8e428f471a1dcce P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false } } }
I20240501 07:59:14.201526 9055 raft_consensus.cc:2932] T d6c28cd91f624673b8e428f471a1dcce P a505e7cb133c4392bfda8a4a20874ae9 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false } } }
I20240501 07:59:14.204144 6172 catalog_manager.cc:5484] T d6c28cd91f624673b8e428f471a1dcce P 8315d22c56f7439a9e1fc82f153f018c reported cstate change: config changed from index 5 to 6, VOTER 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false } } }
I20240501 07:59:14.209765 6295 tablet_service.cc:1514] Processing DeleteTablet for tablet d6c28cd91f624673b8e428f471a1dcce with delete_type TABLET_DATA_TOMBSTONED (TS 8b8b3ad3aa8f4053b2167dd550f2086f not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:45480
I20240501 07:59:14.210186 12008 tablet_replica.cc:330] T d6c28cd91f624673b8e428f471a1dcce P 8b8b3ad3aa8f4053b2167dd550f2086f: stopping tablet replica
I20240501 07:59:14.210320 12008 raft_consensus.cc:2229] T d6c28cd91f624673b8e428f471a1dcce P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:14.210472 12008 raft_consensus.cc:2258] T d6c28cd91f624673b8e428f471a1dcce P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:14.211490 12008 ts_tablet_manager.cc:1875] T d6c28cd91f624673b8e428f471a1dcce P 8b8b3ad3aa8f4053b2167dd550f2086f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:59:14.213583 6160 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d6c28cd91f624673b8e428f471a1dcce with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 07:59:14.213824 12008 ts_tablet_manager.cc:1888] T d6c28cd91f624673b8e428f471a1dcce P 8b8b3ad3aa8f4053b2167dd550f2086f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 07:59:14.213902 12008 log.cc:1196] T d6c28cd91f624673b8e428f471a1dcce P 8b8b3ad3aa8f4053b2167dd550f2086f: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/d6c28cd91f624673b8e428f471a1dcce
I20240501 07:59:14.214483 6157 catalog_manager.cc:4830] TS 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129:46875): tablet d6c28cd91f624673b8e428f471a1dcce (table rebalance_test_table_11 [id=3e062fd8d58941efbd3d190c28bb0059]) successfully deleted
I20240501 07:59:14.231773 6444 raft_consensus.cc:1261] T 72aa4dc7664b48b7a73d34feba7d44d3 P 8315d22c56f7439a9e1fc82f153f018c [term 2 FOLLOWER]: Refusing update from remote peer a505e7cb133c4392bfda8a4a20874ae9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 07:59:14.232311 11985 consensus_queue.cc:1035] T 72aa4dc7664b48b7a73d34feba7d44d3 P a505e7cb133c4392bfda8a4a20874ae9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:14.236608 8793 raft_consensus.cc:1261] T 72aa4dc7664b48b7a73d34feba7d44d3 P 74077b61d6f54342bf9736b4a9f2eade [term 2 FOLLOWER]: Refusing update from remote peer a505e7cb133c4392bfda8a4a20874ae9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 07:59:14.238237 11453 consensus_queue.cc:1035] T 72aa4dc7664b48b7a73d34feba7d44d3 P a505e7cb133c4392bfda8a4a20874ae9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:14.238622 9055 consensus_queue.cc:237] T 72aa4dc7664b48b7a73d34feba7d44d3 P a505e7cb133c4392bfda8a4a20874ae9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { promote: false } }
I20240501 07:59:14.239585 12011 raft_consensus.cc:482] T 1e06c2e0771541bdb61d1d995c93a543 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:14.239684 12011 raft_consensus.cc:504] T 1e06c2e0771541bdb61d1d995c93a543 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } }
I20240501 07:59:14.240000 8924 raft_consensus.cc:1261] T 72aa4dc7664b48b7a73d34feba7d44d3 P 13d05ebdca8c4f489c597ce550dd773d [term 2 FOLLOWER]: Refusing update from remote peer a505e7cb133c4392bfda8a4a20874ae9: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 07:59:14.240080 12011 leader_election.cc:288] T 1e06c2e0771541bdb61d1d995c93a543 P 8315d22c56f7439a9e1fc82f153f018c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134:40141), a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607)
I20240501 07:59:14.240399 10945 consensus_queue.cc:1035] T 72aa4dc7664b48b7a73d34feba7d44d3 P a505e7cb133c4392bfda8a4a20874ae9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:59:14.241552 6444 raft_consensus.cc:1261] T 72aa4dc7664b48b7a73d34feba7d44d3 P 8315d22c56f7439a9e1fc82f153f018c [term 2 FOLLOWER]: Refusing update from remote peer a505e7cb133c4392bfda8a4a20874ae9: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 07:59:14.242007 11453 consensus_queue.cc:1035] T 72aa4dc7664b48b7a73d34feba7d44d3 P a505e7cb133c4392bfda8a4a20874ae9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:59:14.242528 11453 raft_consensus.cc:2932] T 72aa4dc7664b48b7a73d34feba7d44d3 P a505e7cb133c4392bfda8a4a20874ae9 [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { promote: false } } }
I20240501 07:59:14.243506 9056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e06c2e0771541bdb61d1d995c93a543" candidate_uuid: "8315d22c56f7439a9e1fc82f153f018c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a505e7cb133c4392bfda8a4a20874ae9" is_pre_election: true
I20240501 07:59:14.243635 9056 raft_consensus.cc:2454] T 1e06c2e0771541bdb61d1d995c93a543 P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8315d22c56f7439a9e1fc82f153f018c in term 0.
I20240501 07:59:14.244264 8924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e06c2e0771541bdb61d1d995c93a543" candidate_uuid: "8315d22c56f7439a9e1fc82f153f018c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13d05ebdca8c4f489c597ce550dd773d" is_pre_election: true
I20240501 07:59:14.244415 8924 raft_consensus.cc:2454] T 1e06c2e0771541bdb61d1d995c93a543 P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8315d22c56f7439a9e1fc82f153f018c in term 0.
I20240501 07:59:14.244745 6381 leader_election.cc:302] T 1e06c2e0771541bdb61d1d995c93a543 P 8315d22c56f7439a9e1fc82f153f018c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13d05ebdca8c4f489c597ce550dd773d, 8315d22c56f7439a9e1fc82f153f018c; no voters:
I20240501 07:59:14.245033 12011 raft_consensus.cc:2781] T 1e06c2e0771541bdb61d1d995c93a543 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:14.245100 12011 raft_consensus.cc:482] T 1e06c2e0771541bdb61d1d995c93a543 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:14.245082 6158 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 72aa4dc7664b48b7a73d34feba7d44d3 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 07:59:14.245141 12011 raft_consensus.cc:3037] T 1e06c2e0771541bdb61d1d995c93a543 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:14.245640 6171 catalog_manager.cc:5484] T 72aa4dc7664b48b7a73d34feba7d44d3 P a505e7cb133c4392bfda8a4a20874ae9 reported cstate change: config changed from index 3 to 5, VOTER 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133) evicted. New cstate: current_term: 2 leader_uuid: "a505e7cb133c4392bfda8a4a20874ae9" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 07:59:14.245721 6444 raft_consensus.cc:2932] T 72aa4dc7664b48b7a73d34feba7d44d3 P 8315d22c56f7439a9e1fc82f153f018c [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { promote: false } } }
I20240501 07:59:14.246296 12011 raft_consensus.cc:504] T 1e06c2e0771541bdb61d1d995c93a543 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } }
I20240501 07:59:14.246364 8924 raft_consensus.cc:2932] T 72aa4dc7664b48b7a73d34feba7d44d3 P 13d05ebdca8c4f489c597ce550dd773d [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { promote: false } } }
I20240501 07:59:14.246688 12011 leader_election.cc:288] T 1e06c2e0771541bdb61d1d995c93a543 P 8315d22c56f7439a9e1fc82f153f018c [CANDIDATE]: Term 1 election: Requested vote from peers 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134:40141), a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607)
I20240501 07:59:14.247145 9055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e06c2e0771541bdb61d1d995c93a543" candidate_uuid: "8315d22c56f7439a9e1fc82f153f018c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a505e7cb133c4392bfda8a4a20874ae9"
I20240501 07:59:14.247256 9055 raft_consensus.cc:3037] T 1e06c2e0771541bdb61d1d995c93a543 P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:14.247328 8925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e06c2e0771541bdb61d1d995c93a543" candidate_uuid: "8315d22c56f7439a9e1fc82f153f018c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13d05ebdca8c4f489c597ce550dd773d"
I20240501 07:59:14.247440 8925 raft_consensus.cc:3037] T 1e06c2e0771541bdb61d1d995c93a543 P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:14.248275 9055 raft_consensus.cc:2454] T 1e06c2e0771541bdb61d1d995c93a543 P a505e7cb133c4392bfda8a4a20874ae9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8315d22c56f7439a9e1fc82f153f018c in term 1.
I20240501 07:59:14.248390 8925 raft_consensus.cc:2454] T 1e06c2e0771541bdb61d1d995c93a543 P 13d05ebdca8c4f489c597ce550dd773d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8315d22c56f7439a9e1fc82f153f018c in term 1.
I20240501 07:59:14.250068 6381 leader_election.cc:302] T 1e06c2e0771541bdb61d1d995c93a543 P 8315d22c56f7439a9e1fc82f153f018c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13d05ebdca8c4f489c597ce550dd773d, 8315d22c56f7439a9e1fc82f153f018c; no voters:
I20240501 07:59:14.250165 8757 tablet_service.cc:1514] Processing DeleteTablet for tablet 72aa4dc7664b48b7a73d34feba7d44d3 with delete_type TABLET_DATA_TOMBSTONED (TS 74077b61d6f54342bf9736b4a9f2eade not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:43136
I20240501 07:59:14.250316 11735 raft_consensus.cc:2781] T 1e06c2e0771541bdb61d1d995c93a543 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:14.250403 11735 raft_consensus.cc:686] T 1e06c2e0771541bdb61d1d995c93a543 P 8315d22c56f7439a9e1fc82f153f018c [term 1 LEADER]: Becoming Leader. State: Replica: 8315d22c56f7439a9e1fc82f153f018c, State: Running, Role: LEADER
I20240501 07:59:14.250412 12007 tablet_replica.cc:330] T 72aa4dc7664b48b7a73d34feba7d44d3 P 74077b61d6f54342bf9736b4a9f2eade: stopping tablet replica
I20240501 07:59:14.250559 12007 raft_consensus.cc:2229] T 72aa4dc7664b48b7a73d34feba7d44d3 P 74077b61d6f54342bf9736b4a9f2eade [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:14.250537 11735 consensus_queue.cc:237] T 1e06c2e0771541bdb61d1d995c93a543 P 8315d22c56f7439a9e1fc82f153f018c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } }
I20240501 07:59:14.250686 12007 raft_consensus.cc:2258] T 72aa4dc7664b48b7a73d34feba7d44d3 P 74077b61d6f54342bf9736b4a9f2eade [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:14.251053 11402 raft_consensus.cc:1050] T ff9dc384dbac47dc9795617c64688943 P 1f9cb72be98941b5a8e504cbe34132b9: attempting to promote NON_VOTER 83f20e84af4c40d09a4804b6b26e697f to VOTER
I20240501 07:59:14.251334 12007 ts_tablet_manager.cc:1875] T 72aa4dc7664b48b7a73d34feba7d44d3 P 74077b61d6f54342bf9736b4a9f2eade: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:59:14.251475 11402 consensus_queue.cc:237] T ff9dc384dbac47dc9795617c64688943 P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { replace: true } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false } }
I20240501 07:59:14.253579 8925 raft_consensus.cc:1261] T ff9dc384dbac47dc9795617c64688943 P 13d05ebdca8c4f489c597ce550dd773d [term 2 FOLLOWER]: Refusing update from remote peer 1f9cb72be98941b5a8e504cbe34132b9: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 07:59:14.253613 6313 raft_consensus.cc:1261] T ff9dc384dbac47dc9795617c64688943 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 2 FOLLOWER]: Refusing update from remote peer 1f9cb72be98941b5a8e504cbe34132b9: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 07:59:14.253834 12007 ts_tablet_manager.cc:1888] T 72aa4dc7664b48b7a73d34feba7d44d3 P 74077b61d6f54342bf9736b4a9f2eade: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20240501 07:59:14.253907 12007 log.cc:1196] T 72aa4dc7664b48b7a73d34feba7d44d3 P 74077b61d6f54342bf9736b4a9f2eade: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/72aa4dc7664b48b7a73d34feba7d44d3
I20240501 07:59:14.254052 12048 consensus_queue.cc:1035] T ff9dc384dbac47dc9795617c64688943 P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 07:59:14.254637 9189 raft_consensus.cc:1261] T ff9dc384dbac47dc9795617c64688943 P 83f20e84af4c40d09a4804b6b26e697f [term 2 LEARNER]: Refusing update from remote peer 1f9cb72be98941b5a8e504cbe34132b9: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 07:59:14.254727 12048 consensus_queue.cc:1035] T ff9dc384dbac47dc9795617c64688943 P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 07:59:14.255045 6157 catalog_manager.cc:4830] TS 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133:39859): tablet 72aa4dc7664b48b7a73d34feba7d44d3 (table rebalancer_extra_table_8 [id=7f1dc10bb484406aa39a54213d4854b9]) successfully deleted
I20240501 07:59:14.254967 6171 catalog_manager.cc:5484] T 1e06c2e0771541bdb61d1d995c93a543 P 8315d22c56f7439a9e1fc82f153f018c reported cstate change: term changed from 0 to 1, leader changed from <none> to 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130). New cstate: current_term: 1 leader_uuid: "8315d22c56f7439a9e1fc82f153f018c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } health_report { overall_health: HEALTHY } } }
I20240501 07:59:14.255220 12048 consensus_queue.cc:1035] T ff9dc384dbac47dc9795617c64688943 P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 07:59:14.258036 12048 raft_consensus.cc:2932] T ff9dc384dbac47dc9795617c64688943 P 1f9cb72be98941b5a8e504cbe34132b9 [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { replace: true } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false } } }
I20240501 07:59:14.258880 8925 raft_consensus.cc:2932] T ff9dc384dbac47dc9795617c64688943 P 13d05ebdca8c4f489c597ce550dd773d [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { replace: true } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false } } }
I20240501 07:59:14.259224 8757 tablet_service.cc:1514] Processing DeleteTablet for tablet 72aa4dc7664b48b7a73d34feba7d44d3 with delete_type TABLET_DATA_TOMBSTONED (Replica with old config index 3 (current committed config index is 5)) from {username='slave'} at 127.0.0.1:43136
I20240501 07:59:14.259622 6157 catalog_manager.cc:4830] TS 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133:39859): tablet 72aa4dc7664b48b7a73d34feba7d44d3 (table rebalancer_extra_table_8 [id=7f1dc10bb484406aa39a54213d4854b9]) successfully deleted
I20240501 07:59:14.261451 6313 raft_consensus.cc:2932] T ff9dc384dbac47dc9795617c64688943 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { replace: true } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false } } }
I20240501 07:59:14.261796 6165 catalog_manager.cc:5484] T ff9dc384dbac47dc9795617c64688943 P 13d05ebdca8c4f489c597ce550dd773d reported cstate change: config changed from index 8 to 9, 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "1f9cb72be98941b5a8e504cbe34132b9" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { replace: true } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false } } }
I20240501 07:59:14.264221 9189 raft_consensus.cc:2932] T ff9dc384dbac47dc9795617c64688943 P 83f20e84af4c40d09a4804b6b26e697f [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { replace: true } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false } } }
I20240501 07:59:14.271621 6313 consensus_queue.cc:237] T 6c3c0d6fe41f4ae196ae29beb9b68125 P 8b8b3ad3aa8f4053b2167dd550f2086f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: true } }
I20240501 07:59:14.273294 9055 raft_consensus.cc:1261] T 6c3c0d6fe41f4ae196ae29beb9b68125 P a505e7cb133c4392bfda8a4a20874ae9 [term 1 FOLLOWER]: Refusing update from remote peer 8b8b3ad3aa8f4053b2167dd550f2086f: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 07:59:14.273608 6577 raft_consensus.cc:1261] T 6c3c0d6fe41f4ae196ae29beb9b68125 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Refusing update from remote peer 8b8b3ad3aa8f4053b2167dd550f2086f: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 07:59:14.273960 11094 consensus_queue.cc:1035] T 6c3c0d6fe41f4ae196ae29beb9b68125 P 8b8b3ad3aa8f4053b2167dd550f2086f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 07:59:14.274338 11094 consensus_queue.cc:1035] T 6c3c0d6fe41f4ae196ae29beb9b68125 P 8b8b3ad3aa8f4053b2167dd550f2086f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240501 07:59:14.274523 6250 consensus_peers.cc:485] T 6c3c0d6fe41f4ae196ae29beb9b68125 P 8b8b3ad3aa8f4053b2167dd550f2086f -> Peer 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134:40141): Couldn't send request to peer 13d05ebdca8c4f489c597ce550dd773d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6c3c0d6fe41f4ae196ae29beb9b68125. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:14.276355 11901 raft_consensus.cc:2932] T 6c3c0d6fe41f4ae196ae29beb9b68125 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: true } } }
I20240501 07:59:14.277453 6577 raft_consensus.cc:2932] T 6c3c0d6fe41f4ae196ae29beb9b68125 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: true } } }
I20240501 07:59:14.277581 9055 raft_consensus.cc:2932] T 6c3c0d6fe41f4ae196ae29beb9b68125 P a505e7cb133c4392bfda8a4a20874ae9 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: true } } }
I20240501 07:59:14.280248 6172 catalog_manager.cc:5484] T 6c3c0d6fe41f4ae196ae29beb9b68125 P 8b8b3ad3aa8f4053b2167dd550f2086f reported cstate change: config changed from index 6 to 7, NON_VOTER 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 07:59:14.289539 6577 consensus_queue.cc:237] T 114b6dbafa1c4c44ac42f1d2293e3125 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: true } }
I20240501 07:59:14.291239 6444 raft_consensus.cc:1261] T 114b6dbafa1c4c44ac42f1d2293e3125 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Refusing update from remote peer 3b164cbbc00f4e8eb3254ddd349a3ff9: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240501 07:59:14.291299 6510 consensus_peers.cc:485] T 114b6dbafa1c4c44ac42f1d2293e3125 P 3b164cbbc00f4e8eb3254ddd349a3ff9 -> Peer 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133:39859): Couldn't send request to peer 74077b61d6f54342bf9736b4a9f2eade. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 114b6dbafa1c4c44ac42f1d2293e3125. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:14.291446 9055 raft_consensus.cc:1261] T 114b6dbafa1c4c44ac42f1d2293e3125 P a505e7cb133c4392bfda8a4a20874ae9 [term 1 FOLLOWER]: Refusing update from remote peer 3b164cbbc00f4e8eb3254ddd349a3ff9: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 07:59:14.291887 10046 consensus_queue.cc:1035] T 114b6dbafa1c4c44ac42f1d2293e3125 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 07:59:14.292214 12119 consensus_queue.cc:1035] T 114b6dbafa1c4c44ac42f1d2293e3125 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 07:59:14.294173 11119 raft_consensus.cc:2932] T 114b6dbafa1c4c44ac42f1d2293e3125 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: true } } }
I20240501 07:59:14.294572 9055 raft_consensus.cc:2932] T 114b6dbafa1c4c44ac42f1d2293e3125 P a505e7cb133c4392bfda8a4a20874ae9 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: true } } }
I20240501 07:59:14.294878 6444 raft_consensus.cc:2932] T 114b6dbafa1c4c44ac42f1d2293e3125 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: true } } }
I20240501 07:59:14.297295 6165 catalog_manager.cc:5484] T 114b6dbafa1c4c44ac42f1d2293e3125 P a505e7cb133c4392bfda8a4a20874ae9 reported cstate change: config changed from index 6 to 7, NON_VOTER 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: true } } }
I20240501 07:59:14.302783 10046 raft_consensus.cc:1050] T dc550f54e8024d47b1955905e7858a1e P 3b164cbbc00f4e8eb3254ddd349a3ff9: attempting to promote NON_VOTER 74077b61d6f54342bf9736b4a9f2eade to VOTER
I20240501 07:59:14.303197 10046 consensus_queue.cc:237] T dc550f54e8024d47b1955905e7858a1e P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } }
I20240501 07:59:14.305069 6708 raft_consensus.cc:1261] T dc550f54e8024d47b1955905e7858a1e P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Refusing update from remote peer 3b164cbbc00f4e8eb3254ddd349a3ff9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:59:14.305069 8794 raft_consensus.cc:1261] T dc550f54e8024d47b1955905e7858a1e P 74077b61d6f54342bf9736b4a9f2eade [term 1 LEARNER]: Refusing update from remote peer 3b164cbbc00f4e8eb3254ddd349a3ff9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:59:14.305150 6444 raft_consensus.cc:1261] T dc550f54e8024d47b1955905e7858a1e P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Refusing update from remote peer 3b164cbbc00f4e8eb3254ddd349a3ff9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:59:14.305497 11119 consensus_queue.cc:1035] T dc550f54e8024d47b1955905e7858a1e P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:59:14.305833 11119 consensus_queue.cc:1035] T dc550f54e8024d47b1955905e7858a1e P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:59:14.307149 12048 consensus_queue.cc:1035] T a29720541b7b4a79aaa810592ef7c4b1 P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:14.307559 11107 consensus_queue.cc:1035] T dc550f54e8024d47b1955905e7858a1e P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:59:14.308604 11119 raft_consensus.cc:2932] T dc550f54e8024d47b1955905e7858a1e P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } } }
I20240501 07:59:14.310614 8794 raft_consensus.cc:2932] T dc550f54e8024d47b1955905e7858a1e P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } } }
I20240501 07:59:14.311733 11719 consensus_queue.cc:1035] T a29720541b7b4a79aaa810592ef7c4b1 P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:14.312415 6708 raft_consensus.cc:2932] T dc550f54e8024d47b1955905e7858a1e P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } } }
I20240501 07:59:14.312641 6444 raft_consensus.cc:2932] T dc550f54e8024d47b1955905e7858a1e P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } } }
I20240501 07:59:14.313665 8794 consensus_queue.cc:237] T 2f9e82c3f1df4d1292e37e9cea07186e P 74077b61d6f54342bf9736b4a9f2eade [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { replace: true } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { promote: true } }
I20240501 07:59:14.314285 6172 catalog_manager.cc:5484] T dc550f54e8024d47b1955905e7858a1e P 74077b61d6f54342bf9736b4a9f2eade reported cstate change: config changed from index 4 to 5, 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } } }
I20240501 07:59:14.315569 8925 raft_consensus.cc:1261] T 2f9e82c3f1df4d1292e37e9cea07186e P 13d05ebdca8c4f489c597ce550dd773d [term 1 FOLLOWER]: Refusing update from remote peer 74077b61d6f54342bf9736b4a9f2eade: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 07:59:14.316300 8729 consensus_peers.cc:485] T 2f9e82c3f1df4d1292e37e9cea07186e P 74077b61d6f54342bf9736b4a9f2eade -> Peer 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130:32941): Couldn't send request to peer 8315d22c56f7439a9e1fc82f153f018c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2f9e82c3f1df4d1292e37e9cea07186e. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:14.316421 6313 raft_consensus.cc:1261] T 2f9e82c3f1df4d1292e37e9cea07186e P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Refusing update from remote peer 74077b61d6f54342bf9736b4a9f2eade: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:14.316501 11723 consensus_queue.cc:1035] T 2f9e82c3f1df4d1292e37e9cea07186e P 74077b61d6f54342bf9736b4a9f2eade [LEADER]: Connected to new peer: Peer: permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 07:59:14.316849 11724 consensus_queue.cc:1035] T 2f9e82c3f1df4d1292e37e9cea07186e P 74077b61d6f54342bf9736b4a9f2eade [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 07:59:14.319298 11723 raft_consensus.cc:2932] T 2f9e82c3f1df4d1292e37e9cea07186e P 74077b61d6f54342bf9736b4a9f2eade [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { replace: true } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { promote: true } } }
I20240501 07:59:14.319762 6313 raft_consensus.cc:2932] T 2f9e82c3f1df4d1292e37e9cea07186e P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { replace: true } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { promote: true } } }
I20240501 07:59:14.324033 6172 catalog_manager.cc:5484] T 2f9e82c3f1df4d1292e37e9cea07186e P 8b8b3ad3aa8f4053b2167dd550f2086f reported cstate change: config changed from index -1 to 2, NON_VOTER 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130) added. New cstate: current_term: 1 leader_uuid: "74077b61d6f54342bf9736b4a9f2eade" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { replace: true } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { promote: true } } }
I20240501 07:59:14.324275 6577 consensus_queue.cc:237] T dc550f54e8024d47b1955905e7858a1e P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } }
I20240501 07:59:14.325919 6708 raft_consensus.cc:1261] T dc550f54e8024d47b1955905e7858a1e P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Refusing update from remote peer 3b164cbbc00f4e8eb3254ddd349a3ff9: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 07:59:14.326362 11107 consensus_queue.cc:1035] T dc550f54e8024d47b1955905e7858a1e P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 07:59:14.326685 8925 raft_consensus.cc:2932] T 2f9e82c3f1df4d1292e37e9cea07186e P 13d05ebdca8c4f489c597ce550dd773d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { replace: true } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { promote: true } } }
I20240501 07:59:14.326980 8794 raft_consensus.cc:1261] T dc550f54e8024d47b1955905e7858a1e P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Refusing update from remote peer 3b164cbbc00f4e8eb3254ddd349a3ff9: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 07:59:14.327373 11107 consensus_queue.cc:1035] T dc550f54e8024d47b1955905e7858a1e P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 07:59:14.329367 12119 raft_consensus.cc:2932] T dc550f54e8024d47b1955905e7858a1e P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } } }
I20240501 07:59:14.329699 8794 raft_consensus.cc:2932] T dc550f54e8024d47b1955905e7858a1e P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } } }
I20240501 07:59:14.330439 6708 raft_consensus.cc:2932] T dc550f54e8024d47b1955905e7858a1e P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } } }
I20240501 07:59:14.331799 6160 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet dc550f54e8024d47b1955905e7858a1e with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 07:59:14.333204 6167 catalog_manager.cc:5484] T dc550f54e8024d47b1955905e7858a1e P 3b164cbbc00f4e8eb3254ddd349a3ff9 reported cstate change: config changed from index 5 to 6, VOTER 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 07:59:14.335798 6424 tablet_service.cc:1514] Processing DeleteTablet for tablet dc550f54e8024d47b1955905e7858a1e with delete_type TABLET_DATA_TOMBSTONED (TS 8315d22c56f7439a9e1fc82f153f018c not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:45034
I20240501 07:59:14.336103 12057 tablet_replica.cc:330] T dc550f54e8024d47b1955905e7858a1e P 8315d22c56f7439a9e1fc82f153f018c: stopping tablet replica
I20240501 07:59:14.336282 12057 raft_consensus.cc:2229] T dc550f54e8024d47b1955905e7858a1e P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:14.336478 12057 raft_consensus.cc:2258] T dc550f54e8024d47b1955905e7858a1e P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:14.337360 10945 raft_consensus.cc:1050] T f88fe033bce14bf0b1b63a52a7ba137e P a505e7cb133c4392bfda8a4a20874ae9: attempting to promote NON_VOTER 8315d22c56f7439a9e1fc82f153f018c to VOTER
I20240501 07:59:14.337783 10945 consensus_queue.cc:237] T f88fe033bce14bf0b1b63a52a7ba137e P a505e7cb133c4392bfda8a4a20874ae9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { promote: false } }
I20240501 07:59:14.337967 12057 ts_tablet_manager.cc:1875] T dc550f54e8024d47b1955905e7858a1e P 8315d22c56f7439a9e1fc82f153f018c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:59:14.339354 9189 raft_consensus.cc:1261] T f88fe033bce14bf0b1b63a52a7ba137e P 83f20e84af4c40d09a4804b6b26e697f [term 1 FOLLOWER]: Refusing update from remote peer a505e7cb133c4392bfda8a4a20874ae9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 07:59:14.339489 6444 raft_consensus.cc:1261] T f88fe033bce14bf0b1b63a52a7ba137e P 8315d22c56f7439a9e1fc82f153f018c [term 1 LEARNER]: Refusing update from remote peer a505e7cb133c4392bfda8a4a20874ae9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 07:59:14.339491 8925 raft_consensus.cc:1261] T f88fe033bce14bf0b1b63a52a7ba137e P 13d05ebdca8c4f489c597ce550dd773d [term 1 FOLLOWER]: Refusing update from remote peer a505e7cb133c4392bfda8a4a20874ae9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 07:59:14.339929 11453 consensus_queue.cc:1035] T f88fe033bce14bf0b1b63a52a7ba137e P a505e7cb133c4392bfda8a4a20874ae9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 07:59:14.340086 11985 consensus_queue.cc:1035] T f88fe033bce14bf0b1b63a52a7ba137e P a505e7cb133c4392bfda8a4a20874ae9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 07:59:14.340368 11985 consensus_queue.cc:1035] T f88fe033bce14bf0b1b63a52a7ba137e P a505e7cb133c4392bfda8a4a20874ae9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 07:59:14.340487 12057 ts_tablet_manager.cc:1888] T dc550f54e8024d47b1955905e7858a1e P 8315d22c56f7439a9e1fc82f153f018c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 07:59:14.340556 12057 log.cc:1196] T dc550f54e8024d47b1955905e7858a1e P 8315d22c56f7439a9e1fc82f153f018c: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/dc550f54e8024d47b1955905e7858a1e
I20240501 07:59:14.341271 6159 catalog_manager.cc:4830] TS 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130:32941): tablet dc550f54e8024d47b1955905e7858a1e (table rebalance_test_table_17 [id=4de37f887eb14e2dbe5d1609df1e538f]) successfully deleted
I20240501 07:59:14.341360 9055 consensus_queue.cc:237] T 28eaf22342ff4b5394f958d0b659273c P a505e7cb133c4392bfda8a4a20874ae9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: true } }
I20240501 07:59:14.342754 8794 raft_consensus.cc:1261] T 28eaf22342ff4b5394f958d0b659273c P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Refusing update from remote peer a505e7cb133c4392bfda8a4a20874ae9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:14.343108 6708 raft_consensus.cc:1261] T 28eaf22342ff4b5394f958d0b659273c P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Refusing update from remote peer a505e7cb133c4392bfda8a4a20874ae9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 07:59:14.343129 8992 consensus_peers.cc:485] T 28eaf22342ff4b5394f958d0b659273c P a505e7cb133c4392bfda8a4a20874ae9 -> Peer 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131:34471): Couldn't send request to peer 3b164cbbc00f4e8eb3254ddd349a3ff9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 28eaf22342ff4b5394f958d0b659273c. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:14.343151 10945 consensus_queue.cc:1035] T 28eaf22342ff4b5394f958d0b659273c P a505e7cb133c4392bfda8a4a20874ae9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 07:59:14.343693 10945 consensus_queue.cc:1035] T 28eaf22342ff4b5394f958d0b659273c P a505e7cb133c4392bfda8a4a20874ae9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 07:59:14.344033 10945 raft_consensus.cc:2932] T f88fe033bce14bf0b1b63a52a7ba137e P a505e7cb133c4392bfda8a4a20874ae9 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { promote: false } } }
I20240501 07:59:14.344805 6444 raft_consensus.cc:2932] T f88fe033bce14bf0b1b63a52a7ba137e P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { promote: false } } }
I20240501 07:59:14.345427 12192 raft_consensus.cc:2932] T 28eaf22342ff4b5394f958d0b659273c P a505e7cb133c4392bfda8a4a20874ae9 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: true } } }
I20240501 07:59:14.346365 9189 raft_consensus.cc:2932] T f88fe033bce14bf0b1b63a52a7ba137e P 83f20e84af4c40d09a4804b6b26e697f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { promote: false } } }
I20240501 07:59:14.347285 6708 raft_consensus.cc:2932] T 28eaf22342ff4b5394f958d0b659273c P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: true } } }
I20240501 07:59:14.347494 8794 raft_consensus.cc:2932] T 28eaf22342ff4b5394f958d0b659273c P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: true } } }
I20240501 07:59:14.347744 6165 catalog_manager.cc:5484] T f88fe033bce14bf0b1b63a52a7ba137e P 8315d22c56f7439a9e1fc82f153f018c reported cstate change: config changed from index 2 to 3, 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a505e7cb133c4392bfda8a4a20874ae9" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { promote: false } } }
I20240501 07:59:14.348809 12159 consensus_queue.cc:1035] T 1c32cd00bc344ae1b161d3f9b43b53d3 P 8b8b3ad3aa8f4053b2167dd550f2086f [LEADER]: Connected to new peer: Peer: permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:14.350510 8925 raft_consensus.cc:2932] T f88fe033bce14bf0b1b63a52a7ba137e P 13d05ebdca8c4f489c597ce550dd773d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { promote: false } } }
I20240501 07:59:14.351596 6167 catalog_manager.cc:5484] T 28eaf22342ff4b5394f958d0b659273c P 1f9cb72be98941b5a8e504cbe34132b9 reported cstate change: config changed from index -1 to 2, NON_VOTER 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131) added. New cstate: current_term: 1 leader_uuid: "a505e7cb133c4392bfda8a4a20874ae9" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: true } } }
I20240501 07:59:14.354895 12159 consensus_queue.cc:1035] T 1c32cd00bc344ae1b161d3f9b43b53d3 P 8b8b3ad3aa8f4053b2167dd550f2086f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:14.355683 11107 consensus_queue.cc:1035] T 144ff11d2a504f7ab7a230e6822d24ca P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:14.360085 9055 consensus_queue.cc:237] T f88fe033bce14bf0b1b63a52a7ba137e P a505e7cb133c4392bfda8a4a20874ae9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { promote: false } }
I20240501 07:59:14.362761 6444 raft_consensus.cc:1261] T f88fe033bce14bf0b1b63a52a7ba137e P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Refusing update from remote peer a505e7cb133c4392bfda8a4a20874ae9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:59:14.362896 8925 raft_consensus.cc:1261] T f88fe033bce14bf0b1b63a52a7ba137e P 13d05ebdca8c4f489c597ce550dd773d [term 1 FOLLOWER]: Refusing update from remote peer a505e7cb133c4392bfda8a4a20874ae9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:59:14.363283 12200 consensus_queue.cc:1035] T f88fe033bce14bf0b1b63a52a7ba137e P a505e7cb133c4392bfda8a4a20874ae9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:14.364535 12200 consensus_queue.cc:1035] T f88fe033bce14bf0b1b63a52a7ba137e P a505e7cb133c4392bfda8a4a20874ae9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:14.367285 11107 consensus_queue.cc:1035] T 144ff11d2a504f7ab7a230e6822d24ca P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:14.368046 11985 raft_consensus.cc:2932] T f88fe033bce14bf0b1b63a52a7ba137e P a505e7cb133c4392bfda8a4a20874ae9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { promote: false } } }
I20240501 07:59:14.368541 6444 raft_consensus.cc:2932] T f88fe033bce14bf0b1b63a52a7ba137e P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { promote: false } } }
I20240501 07:59:14.369889 6158 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f88fe033bce14bf0b1b63a52a7ba137e with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 07:59:14.370465 8925 raft_consensus.cc:2932] T f88fe033bce14bf0b1b63a52a7ba137e P 13d05ebdca8c4f489c597ce550dd773d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { promote: false } } }
I20240501 07:59:14.371292 6172 catalog_manager.cc:5484] T f88fe033bce14bf0b1b63a52a7ba137e P a505e7cb133c4392bfda8a4a20874ae9 reported cstate change: config changed from index 3 to 4, VOTER 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136) evicted. New cstate: current_term: 1 leader_uuid: "a505e7cb133c4392bfda8a4a20874ae9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 07:59:14.374109 9169 tablet_service.cc:1514] Processing DeleteTablet for tablet f88fe033bce14bf0b1b63a52a7ba137e with delete_type TABLET_DATA_TOMBSTONED (TS 83f20e84af4c40d09a4804b6b26e697f not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:33594
I20240501 07:59:14.374351 12006 tablet_replica.cc:330] T f88fe033bce14bf0b1b63a52a7ba137e P 83f20e84af4c40d09a4804b6b26e697f: stopping tablet replica
I20240501 07:59:14.374552 12006 raft_consensus.cc:2229] T f88fe033bce14bf0b1b63a52a7ba137e P 83f20e84af4c40d09a4804b6b26e697f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:14.374835 12006 raft_consensus.cc:2258] T f88fe033bce14bf0b1b63a52a7ba137e P 83f20e84af4c40d09a4804b6b26e697f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:14.375540 12006 ts_tablet_manager.cc:1875] T f88fe033bce14bf0b1b63a52a7ba137e P 83f20e84af4c40d09a4804b6b26e697f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:59:14.377792 12006 ts_tablet_manager.cc:1888] T f88fe033bce14bf0b1b63a52a7ba137e P 83f20e84af4c40d09a4804b6b26e697f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 07:59:14.377858 12006 log.cc:1196] T f88fe033bce14bf0b1b63a52a7ba137e P 83f20e84af4c40d09a4804b6b26e697f: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/f88fe033bce14bf0b1b63a52a7ba137e
I20240501 07:59:14.378335 6158 catalog_manager.cc:4830] TS 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): tablet f88fe033bce14bf0b1b63a52a7ba137e (table rebalancer_extra_table_0 [id=9ec2540a0f4b49ada014e0c68ec3b2a0]) successfully deleted
I20240501 07:59:14.383008 6313 raft_consensus.cc:1203] T 6daebcfc41754ec4aca939ca20588854 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 07:59:14.394948 9189 raft_consensus.cc:1203] T 6dc929e2a6ae45bd8b1f297583636bb6 P 83f20e84af4c40d09a4804b6b26e697f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240501 07:59:14.399775 6172 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55950:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 07:59:14.404809 8925 raft_consensus.cc:1261] T 1e06c2e0771541bdb61d1d995c93a543 P 13d05ebdca8c4f489c597ce550dd773d [term 1 FOLLOWER]: Refusing update from remote peer 8315d22c56f7439a9e1fc82f153f018c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:14.404799 9056 raft_consensus.cc:1261] T 1e06c2e0771541bdb61d1d995c93a543 P a505e7cb133c4392bfda8a4a20874ae9 [term 1 FOLLOWER]: Refusing update from remote peer 8315d22c56f7439a9e1fc82f153f018c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:14.405673 11735 consensus_queue.cc:1035] T 1e06c2e0771541bdb61d1d995c93a543 P 8315d22c56f7439a9e1fc82f153f018c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:14.406018 11735 consensus_queue.cc:1035] T 1e06c2e0771541bdb61d1d995c93a543 P 8315d22c56f7439a9e1fc82f153f018c [LEADER]: Connected to new peer: Peer: permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:14.409209 6170 catalog_manager.cc:5576] TS a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607) does not have the latest schema for tablet 1e06c2e0771541bdb61d1d995c93a543 (table rebalancer_dynamic_table [id=b2142804a6254da283ecbc4af2be1c08]). Expected version 1 got 0
I20240501 07:59:14.410209 6172 catalog_manager.cc:5576] TS 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134:40141) does not have the latest schema for tablet 1e06c2e0771541bdb61d1d995c93a543 (table rebalancer_dynamic_table [id=b2142804a6254da283ecbc4af2be1c08]). Expected version 1 got 0
I20240501 07:59:14.409441 12237 tablet.cc:1722] T 1e06c2e0771541bdb61d1d995c93a543 P 8315d22c56f7439a9e1fc82f153f018c: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 07:59:14.411811 6170 catalog_manager.cc:5576] TS 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130:32941) does not have the latest schema for tablet 1e06c2e0771541bdb61d1d995c93a543 (table rebalancer_dynamic_table [id=b2142804a6254da283ecbc4af2be1c08]). Expected version 1 got 0
I20240501 07:59:14.412678 6172 catalog_manager.cc:5576] TS 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134:40141) does not have the latest schema for tablet 1e06c2e0771541bdb61d1d995c93a543 (table rebalancer_dynamic_table [id=b2142804a6254da283ecbc4af2be1c08]). Expected version 1 got 0
I20240501 07:59:14.413465 12237 tablet.cc:1620] T 1e06c2e0771541bdb61d1d995c93a543 P 8315d22c56f7439a9e1fc82f153f018c: MemRowSet was empty: no flush needed.
I20240501 07:59:14.414785 12238 tablet.cc:1722] T 1e06c2e0771541bdb61d1d995c93a543 P 13d05ebdca8c4f489c597ce550dd773d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 07:59:14.415127 12238 tablet.cc:1620] T 1e06c2e0771541bdb61d1d995c93a543 P 13d05ebdca8c4f489c597ce550dd773d: MemRowSet was empty: no flush needed.
I20240501 07:59:14.417685 12239 tablet.cc:1722] T 1e06c2e0771541bdb61d1d995c93a543 P a505e7cb133c4392bfda8a4a20874ae9: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 07:59:14.418530 12239 tablet.cc:1620] T 1e06c2e0771541bdb61d1d995c93a543 P a505e7cb133c4392bfda8a4a20874ae9: MemRowSet was empty: no flush needed.
I20240501 07:59:14.419498 6170 catalog_manager.cc:5576] TS a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607) does not have the latest schema for tablet 1e06c2e0771541bdb61d1d995c93a543 (table rebalancer_dynamic_table [id=b2142804a6254da283ecbc4af2be1c08]). Expected version 1 got 0
I20240501 07:59:14.421362 6172 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 0bc706e85ce74cef97a4827ea4d4aa65: rebalancer_dynamic_table [id=b2142804a6254da283ecbc4af2be1c08] alter complete (version 1)
I20240501 07:59:14.466496 6708 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ff9dc384dbac47dc9795617c64688943"
dest_uuid: "1f9cb72be98941b5a8e504cbe34132b9"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:52682
I20240501 07:59:14.466666 6708 raft_consensus.cc:595] T ff9dc384dbac47dc9795617c64688943 P 1f9cb72be98941b5a8e504cbe34132b9 [term 2 LEADER]: Received request to transfer leadership
I20240501 07:59:14.468045 6170 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55950:
name: "rebalancer_extra_table_18"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 07:59:14.474962 6424 tablet_service.cc:1467] Processing CreateTablet for tablet bab84387a40d40c3b8af7f09eb462886 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=b2866453baf645358629862b98006fa6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:14.475286 6424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bab84387a40d40c3b8af7f09eb462886. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:14.476245 9167 tablet_service.cc:1467] Processing CreateTablet for tablet 36aaf192042f4320b2526322424dc640 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=b2866453baf645358629862b98006fa6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:14.476547 9167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36aaf192042f4320b2526322424dc640. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:14.476572 9169 tablet_service.cc:1467] Processing CreateTablet for tablet bab84387a40d40c3b8af7f09eb462886 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=b2866453baf645358629862b98006fa6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:14.476845 9169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bab84387a40d40c3b8af7f09eb462886. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:14.476945 8905 tablet_service.cc:1467] Processing CreateTablet for tablet bab84387a40d40c3b8af7f09eb462886 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=b2866453baf645358629862b98006fa6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:14.477272 8905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bab84387a40d40c3b8af7f09eb462886. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:14.478206 8904 tablet_service.cc:1467] Processing CreateTablet for tablet 36aaf192042f4320b2526322424dc640 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=b2866453baf645358629862b98006fa6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:14.478453 8904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36aaf192042f4320b2526322424dc640. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:14.479063 9035 tablet_service.cc:1467] Processing CreateTablet for tablet 0f9e2549fe294a93b3b25081a15b2028 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=b2866453baf645358629862b98006fa6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:14.479353 9035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f9e2549fe294a93b3b25081a15b2028. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:14.479351 11987 tablet_bootstrap.cc:492] T bab84387a40d40c3b8af7f09eb462886 P 13d05ebdca8c4f489c597ce550dd773d: Bootstrap starting.
I20240501 07:59:14.480219 11987 tablet_bootstrap.cc:654] T bab84387a40d40c3b8af7f09eb462886 P 13d05ebdca8c4f489c597ce550dd773d: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:14.480438 12035 tablet_bootstrap.cc:492] T bab84387a40d40c3b8af7f09eb462886 P 8315d22c56f7439a9e1fc82f153f018c: Bootstrap starting.
I20240501 07:59:14.481700 12035 tablet_bootstrap.cc:654] T bab84387a40d40c3b8af7f09eb462886 P 8315d22c56f7439a9e1fc82f153f018c: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:14.481896 8757 tablet_service.cc:1467] Processing CreateTablet for tablet 0f9e2549fe294a93b3b25081a15b2028 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=b2866453baf645358629862b98006fa6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:14.482173 8757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f9e2549fe294a93b3b25081a15b2028. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:14.482174 6313 raft_consensus.cc:1203] T 54b3efabad7f4b608cc9d931546b5743 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 07:59:14.482738 6556 tablet_service.cc:1467] Processing CreateTablet for tablet 0f9e2549fe294a93b3b25081a15b2028 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=b2866453baf645358629862b98006fa6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:14.483016 6556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f9e2549fe294a93b3b25081a15b2028. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:14.484596 8774 tablet_service.cc:1467] Processing CreateTablet for tablet 36aaf192042f4320b2526322424dc640 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=b2866453baf645358629862b98006fa6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:14.484853 8774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36aaf192042f4320b2526322424dc640. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:14.486764 11987 tablet_bootstrap.cc:492] T bab84387a40d40c3b8af7f09eb462886 P 13d05ebdca8c4f489c597ce550dd773d: No bootstrap required, opened a new log
I20240501 07:59:14.486804 12037 tablet_bootstrap.cc:492] T 0f9e2549fe294a93b3b25081a15b2028 P a505e7cb133c4392bfda8a4a20874ae9: Bootstrap starting.
I20240501 07:59:14.487058 12280 tablet_bootstrap.cc:492] T 36aaf192042f4320b2526322424dc640 P 83f20e84af4c40d09a4804b6b26e697f: Bootstrap starting.
I20240501 07:59:14.487828 12037 tablet_bootstrap.cc:654] T 0f9e2549fe294a93b3b25081a15b2028 P a505e7cb133c4392bfda8a4a20874ae9: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:14.488126 12283 tablet_bootstrap.cc:492] T 0f9e2549fe294a93b3b25081a15b2028 P 74077b61d6f54342bf9736b4a9f2eade: Bootstrap starting.
I20240501 07:59:14.488380 11987 ts_tablet_manager.cc:1367] T bab84387a40d40c3b8af7f09eb462886 P 13d05ebdca8c4f489c597ce550dd773d: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240501 07:59:14.488915 11987 raft_consensus.cc:348] T bab84387a40d40c3b8af7f09eb462886 P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } }
I20240501 07:59:14.489113 11987 raft_consensus.cc:374] T bab84387a40d40c3b8af7f09eb462886 P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:14.489156 11987 raft_consensus.cc:729] T bab84387a40d40c3b8af7f09eb462886 P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13d05ebdca8c4f489c597ce550dd773d, State: Initialized, Role: FOLLOWER
I20240501 07:59:14.489183 12283 tablet_bootstrap.cc:654] T 0f9e2549fe294a93b3b25081a15b2028 P 74077b61d6f54342bf9736b4a9f2eade: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:14.489282 11987 consensus_queue.cc:260] T bab84387a40d40c3b8af7f09eb462886 P 13d05ebdca8c4f489c597ce550dd773d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } }
I20240501 07:59:14.489574 11987 ts_tablet_manager.cc:1398] T bab84387a40d40c3b8af7f09eb462886 P 13d05ebdca8c4f489c597ce550dd773d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:14.489648 12280 tablet_bootstrap.cc:654] T 36aaf192042f4320b2526322424dc640 P 83f20e84af4c40d09a4804b6b26e697f: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:14.490801 12037 tablet_bootstrap.cc:492] T 0f9e2549fe294a93b3b25081a15b2028 P a505e7cb133c4392bfda8a4a20874ae9: No bootstrap required, opened a new log
I20240501 07:59:14.490869 12283 tablet_bootstrap.cc:492] T 0f9e2549fe294a93b3b25081a15b2028 P 74077b61d6f54342bf9736b4a9f2eade: No bootstrap required, opened a new log
I20240501 07:59:14.490895 12037 ts_tablet_manager.cc:1367] T 0f9e2549fe294a93b3b25081a15b2028 P a505e7cb133c4392bfda8a4a20874ae9: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 07:59:14.490952 12283 ts_tablet_manager.cc:1367] T 0f9e2549fe294a93b3b25081a15b2028 P 74077b61d6f54342bf9736b4a9f2eade: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240501 07:59:14.491353 12283 raft_consensus.cc:348] T 0f9e2549fe294a93b3b25081a15b2028 P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } }
I20240501 07:59:14.491446 12283 raft_consensus.cc:374] T 0f9e2549fe294a93b3b25081a15b2028 P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:14.491389 12037 raft_consensus.cc:348] T 0f9e2549fe294a93b3b25081a15b2028 P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } }
I20240501 07:59:14.491474 12283 raft_consensus.cc:729] T 0f9e2549fe294a93b3b25081a15b2028 P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74077b61d6f54342bf9736b4a9f2eade, State: Initialized, Role: FOLLOWER
I20240501 07:59:14.491494 12037 raft_consensus.cc:374] T 0f9e2549fe294a93b3b25081a15b2028 P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:14.491528 12037 raft_consensus.cc:729] T 0f9e2549fe294a93b3b25081a15b2028 P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a505e7cb133c4392bfda8a4a20874ae9, State: Initialized, Role: FOLLOWER
I20240501 07:59:14.491636 12037 consensus_queue.cc:260] T 0f9e2549fe294a93b3b25081a15b2028 P a505e7cb133c4392bfda8a4a20874ae9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } }
I20240501 07:59:14.491890 12037 ts_tablet_manager.cc:1398] T 0f9e2549fe294a93b3b25081a15b2028 P a505e7cb133c4392bfda8a4a20874ae9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:14.491654 12283 consensus_queue.cc:260] T 0f9e2549fe294a93b3b25081a15b2028 P 74077b61d6f54342bf9736b4a9f2eade [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } }
I20240501 07:59:14.493602 12283 ts_tablet_manager.cc:1398] T 0f9e2549fe294a93b3b25081a15b2028 P 74077b61d6f54342bf9736b4a9f2eade: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 07:59:14.494755 12283 tablet_bootstrap.cc:492] T 36aaf192042f4320b2526322424dc640 P 74077b61d6f54342bf9736b4a9f2eade: Bootstrap starting.
I20240501 07:59:14.495839 12283 tablet_bootstrap.cc:654] T 36aaf192042f4320b2526322424dc640 P 74077b61d6f54342bf9736b4a9f2eade: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:14.497413 11987 tablet_bootstrap.cc:492] T 36aaf192042f4320b2526322424dc640 P 13d05ebdca8c4f489c597ce550dd773d: Bootstrap starting.
I20240501 07:59:14.497725 12035 tablet_bootstrap.cc:492] T bab84387a40d40c3b8af7f09eb462886 P 8315d22c56f7439a9e1fc82f153f018c: No bootstrap required, opened a new log
I20240501 07:59:14.497833 12035 ts_tablet_manager.cc:1367] T bab84387a40d40c3b8af7f09eb462886 P 8315d22c56f7439a9e1fc82f153f018c: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.000s
I20240501 07:59:14.498310 12283 tablet_bootstrap.cc:492] T 36aaf192042f4320b2526322424dc640 P 74077b61d6f54342bf9736b4a9f2eade: No bootstrap required, opened a new log
I20240501 07:59:14.498386 12283 ts_tablet_manager.cc:1367] T 36aaf192042f4320b2526322424dc640 P 74077b61d6f54342bf9736b4a9f2eade: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 07:59:14.498376 12035 raft_consensus.cc:348] T bab84387a40d40c3b8af7f09eb462886 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } }
I20240501 07:59:14.498514 12035 raft_consensus.cc:374] T bab84387a40d40c3b8af7f09eb462886 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:14.498548 12035 raft_consensus.cc:729] T bab84387a40d40c3b8af7f09eb462886 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8315d22c56f7439a9e1fc82f153f018c, State: Initialized, Role: FOLLOWER
I20240501 07:59:14.498669 12035 consensus_queue.cc:260] T bab84387a40d40c3b8af7f09eb462886 P 8315d22c56f7439a9e1fc82f153f018c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } }
I20240501 07:59:14.498836 12283 raft_consensus.cc:348] T 36aaf192042f4320b2526322424dc640 P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } }
I20240501 07:59:14.498957 12283 raft_consensus.cc:374] T 36aaf192042f4320b2526322424dc640 P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:14.498960 12035 ts_tablet_manager.cc:1398] T bab84387a40d40c3b8af7f09eb462886 P 8315d22c56f7439a9e1fc82f153f018c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:14.498977 11987 tablet_bootstrap.cc:654] T 36aaf192042f4320b2526322424dc640 P 13d05ebdca8c4f489c597ce550dd773d: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:14.498992 12283 raft_consensus.cc:729] T 36aaf192042f4320b2526322424dc640 P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74077b61d6f54342bf9736b4a9f2eade, State: Initialized, Role: FOLLOWER
I20240501 07:59:14.499125 12283 consensus_queue.cc:260] T 36aaf192042f4320b2526322424dc640 P 74077b61d6f54342bf9736b4a9f2eade [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } }
I20240501 07:59:14.499675 12283 ts_tablet_manager.cc:1398] T 36aaf192042f4320b2526322424dc640 P 74077b61d6f54342bf9736b4a9f2eade: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:14.499894 12284 tablet_bootstrap.cc:492] T 0f9e2549fe294a93b3b25081a15b2028 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Bootstrap starting.
I20240501 07:59:14.500891 12284 tablet_bootstrap.cc:654] T 0f9e2549fe294a93b3b25081a15b2028 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:14.501612 11987 tablet_bootstrap.cc:492] T 36aaf192042f4320b2526322424dc640 P 13d05ebdca8c4f489c597ce550dd773d: No bootstrap required, opened a new log
I20240501 07:59:14.501708 11987 ts_tablet_manager.cc:1367] T 36aaf192042f4320b2526322424dc640 P 13d05ebdca8c4f489c597ce550dd773d: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 07:59:14.501808 12280 tablet_bootstrap.cc:492] T 36aaf192042f4320b2526322424dc640 P 83f20e84af4c40d09a4804b6b26e697f: No bootstrap required, opened a new log
I20240501 07:59:14.501900 12280 ts_tablet_manager.cc:1367] T 36aaf192042f4320b2526322424dc640 P 83f20e84af4c40d09a4804b6b26e697f: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.000s
I20240501 07:59:14.502121 11987 raft_consensus.cc:348] T 36aaf192042f4320b2526322424dc640 P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } }
I20240501 07:59:14.502207 11987 raft_consensus.cc:374] T 36aaf192042f4320b2526322424dc640 P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:14.502226 11987 raft_consensus.cc:729] T 36aaf192042f4320b2526322424dc640 P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13d05ebdca8c4f489c597ce550dd773d, State: Initialized, Role: FOLLOWER
I20240501 07:59:14.502333 11987 consensus_queue.cc:260] T 36aaf192042f4320b2526322424dc640 P 13d05ebdca8c4f489c597ce550dd773d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } }
I20240501 07:59:14.502332 12280 raft_consensus.cc:348] T 36aaf192042f4320b2526322424dc640 P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } }
I20240501 07:59:14.502429 12280 raft_consensus.cc:374] T 36aaf192042f4320b2526322424dc640 P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:14.502455 12280 raft_consensus.cc:729] T 36aaf192042f4320b2526322424dc640 P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83f20e84af4c40d09a4804b6b26e697f, State: Initialized, Role: FOLLOWER
I20240501 07:59:14.502559 12280 consensus_queue.cc:260] T 36aaf192042f4320b2526322424dc640 P 83f20e84af4c40d09a4804b6b26e697f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } }
I20240501 07:59:14.502609 11987 ts_tablet_manager.cc:1398] T 36aaf192042f4320b2526322424dc640 P 13d05ebdca8c4f489c597ce550dd773d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:14.502789 12280 ts_tablet_manager.cc:1398] T 36aaf192042f4320b2526322424dc640 P 83f20e84af4c40d09a4804b6b26e697f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:59:14.502938 12280 tablet_bootstrap.cc:492] T bab84387a40d40c3b8af7f09eb462886 P 83f20e84af4c40d09a4804b6b26e697f: Bootstrap starting.
I20240501 07:59:14.503803 12280 tablet_bootstrap.cc:654] T bab84387a40d40c3b8af7f09eb462886 P 83f20e84af4c40d09a4804b6b26e697f: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:14.508474 12284 tablet_bootstrap.cc:492] T 0f9e2549fe294a93b3b25081a15b2028 P 3b164cbbc00f4e8eb3254ddd349a3ff9: No bootstrap required, opened a new log
I20240501 07:59:14.508585 12284 ts_tablet_manager.cc:1367] T 0f9e2549fe294a93b3b25081a15b2028 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240501 07:59:14.509040 12284 raft_consensus.cc:348] T 0f9e2549fe294a93b3b25081a15b2028 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } }
I20240501 07:59:14.509145 12284 raft_consensus.cc:374] T 0f9e2549fe294a93b3b25081a15b2028 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:14.509182 12284 raft_consensus.cc:729] T 0f9e2549fe294a93b3b25081a15b2028 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b164cbbc00f4e8eb3254ddd349a3ff9, State: Initialized, Role: FOLLOWER
I20240501 07:59:14.509321 12284 consensus_queue.cc:260] T 0f9e2549fe294a93b3b25081a15b2028 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } }
I20240501 07:59:14.509613 12280 tablet_bootstrap.cc:492] T bab84387a40d40c3b8af7f09eb462886 P 83f20e84af4c40d09a4804b6b26e697f: No bootstrap required, opened a new log
I20240501 07:59:14.509681 12284 ts_tablet_manager.cc:1398] T 0f9e2549fe294a93b3b25081a15b2028 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:14.509707 12280 ts_tablet_manager.cc:1367] T bab84387a40d40c3b8af7f09eb462886 P 83f20e84af4c40d09a4804b6b26e697f: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240501 07:59:14.510087 12280 raft_consensus.cc:348] T bab84387a40d40c3b8af7f09eb462886 P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } }
I20240501 07:59:14.510183 12280 raft_consensus.cc:374] T bab84387a40d40c3b8af7f09eb462886 P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:14.510218 12280 raft_consensus.cc:729] T bab84387a40d40c3b8af7f09eb462886 P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83f20e84af4c40d09a4804b6b26e697f, State: Initialized, Role: FOLLOWER
I20240501 07:59:14.510356 12280 consensus_queue.cc:260] T bab84387a40d40c3b8af7f09eb462886 P 83f20e84af4c40d09a4804b6b26e697f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } }
I20240501 07:59:14.510556 12280 ts_tablet_manager.cc:1398] T bab84387a40d40c3b8af7f09eb462886 P 83f20e84af4c40d09a4804b6b26e697f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:59:14.523226 6172 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55950:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 07:59:14.523363 6708 consensus_queue.cc:237] T 6221deb431d94bfca1c14696e0e0a0c5 P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: true } }
I20240501 07:59:14.525177 6577 raft_consensus.cc:1261] T 6221deb431d94bfca1c14696e0e0a0c5 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Refusing update from remote peer 1f9cb72be98941b5a8e504cbe34132b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:59:14.525573 6444 raft_consensus.cc:1261] T 6221deb431d94bfca1c14696e0e0a0c5 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Refusing update from remote peer 1f9cb72be98941b5a8e504cbe34132b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 07:59:14.525873 6641 consensus_peers.cc:485] T 6221deb431d94bfca1c14696e0e0a0c5 P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133:39859): Couldn't send request to peer 74077b61d6f54342bf9736b4a9f2eade. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6221deb431d94bfca1c14696e0e0a0c5. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:14.526235 12049 consensus_queue.cc:1035] T 6221deb431d94bfca1c14696e0e0a0c5 P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:14.527055 12049 consensus_queue.cc:1035] T 6221deb431d94bfca1c14696e0e0a0c5 P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:14.529150 10324 raft_consensus.cc:2932] T 6221deb431d94bfca1c14696e0e0a0c5 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: true } } }
I20240501 07:59:14.529574 6444 raft_consensus.cc:2932] T 6221deb431d94bfca1c14696e0e0a0c5 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: true } } }
I20240501 07:59:14.530651 12237 tablet.cc:1722] T 1e06c2e0771541bdb61d1d995c93a543 P 8315d22c56f7439a9e1fc82f153f018c: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 07:59:14.530978 12237 tablet.cc:1620] T 1e06c2e0771541bdb61d1d995c93a543 P 8315d22c56f7439a9e1fc82f153f018c: MemRowSet was empty: no flush needed.
I20240501 07:59:14.531211 12239 tablet.cc:1722] T 1e06c2e0771541bdb61d1d995c93a543 P a505e7cb133c4392bfda8a4a20874ae9: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 07:59:14.531514 12239 tablet.cc:1620] T 1e06c2e0771541bdb61d1d995c93a543 P a505e7cb133c4392bfda8a4a20874ae9: MemRowSet was empty: no flush needed.
I20240501 07:59:14.532881 6171 catalog_manager.cc:5484] T 6221deb431d94bfca1c14696e0e0a0c5 P 1f9cb72be98941b5a8e504cbe34132b9 reported cstate change: config changed from index -1 to 4, NON_VOTER 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "1f9cb72be98941b5a8e504cbe34132b9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 07:59:14.535511 12238 tablet.cc:1722] T 1e06c2e0771541bdb61d1d995c93a543 P 13d05ebdca8c4f489c597ce550dd773d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 07:59:14.535851 12238 tablet.cc:1620] T 1e06c2e0771541bdb61d1d995c93a543 P 13d05ebdca8c4f489c597ce550dd773d: MemRowSet was empty: no flush needed.
I20240501 07:59:14.536736 6577 raft_consensus.cc:2932] T 6221deb431d94bfca1c14696e0e0a0c5 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: true } } }
I20240501 07:59:14.541251 6172 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 0bc706e85ce74cef97a4827ea4d4aa65: rebalancer_dynamic_table [id=b2142804a6254da283ecbc4af2be1c08] alter complete (version 2)
I20240501 07:59:14.543723 11723 raft_consensus.cc:482] T 0f9e2549fe294a93b3b25081a15b2028 P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:14.543850 11723 raft_consensus.cc:504] T 0f9e2549fe294a93b3b25081a15b2028 P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } }
I20240501 07:59:14.544697 11723 leader_election.cc:288] T 0f9e2549fe294a93b3b25081a15b2028 P 74077b61d6f54342bf9736b4a9f2eade [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607), 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131:34471)
I20240501 07:59:14.545070 6577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f9e2549fe294a93b3b25081a15b2028" candidate_uuid: "74077b61d6f54342bf9736b4a9f2eade" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" is_pre_election: true
I20240501 07:59:14.545061 9056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f9e2549fe294a93b3b25081a15b2028" candidate_uuid: "74077b61d6f54342bf9736b4a9f2eade" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a505e7cb133c4392bfda8a4a20874ae9" is_pre_election: true
I20240501 07:59:14.545212 6577 raft_consensus.cc:2454] T 0f9e2549fe294a93b3b25081a15b2028 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74077b61d6f54342bf9736b4a9f2eade in term 0.
I20240501 07:59:14.545219 9056 raft_consensus.cc:2454] T 0f9e2549fe294a93b3b25081a15b2028 P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74077b61d6f54342bf9736b4a9f2eade in term 0.
I20240501 07:59:14.545464 8730 leader_election.cc:302] T 0f9e2549fe294a93b3b25081a15b2028 P 74077b61d6f54342bf9736b4a9f2eade [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b164cbbc00f4e8eb3254ddd349a3ff9, 74077b61d6f54342bf9736b4a9f2eade; no voters:
I20240501 07:59:14.545776 11723 raft_consensus.cc:2781] T 0f9e2549fe294a93b3b25081a15b2028 P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:14.545842 11723 raft_consensus.cc:482] T 0f9e2549fe294a93b3b25081a15b2028 P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:14.545888 11723 raft_consensus.cc:3037] T 0f9e2549fe294a93b3b25081a15b2028 P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:14.546810 11723 raft_consensus.cc:504] T 0f9e2549fe294a93b3b25081a15b2028 P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } }
I20240501 07:59:14.547107 11723 leader_election.cc:288] T 0f9e2549fe294a93b3b25081a15b2028 P 74077b61d6f54342bf9736b4a9f2eade [CANDIDATE]: Term 1 election: Requested vote from peers a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607), 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131:34471)
I20240501 07:59:14.547415 6575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f9e2549fe294a93b3b25081a15b2028" candidate_uuid: "74077b61d6f54342bf9736b4a9f2eade" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9"
I20240501 07:59:14.547535 6575 raft_consensus.cc:3037] T 0f9e2549fe294a93b3b25081a15b2028 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:14.547504 9056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f9e2549fe294a93b3b25081a15b2028" candidate_uuid: "74077b61d6f54342bf9736b4a9f2eade" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a505e7cb133c4392bfda8a4a20874ae9"
I20240501 07:59:14.547605 9056 raft_consensus.cc:3037] T 0f9e2549fe294a93b3b25081a15b2028 P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:14.548529 6575 raft_consensus.cc:2454] T 0f9e2549fe294a93b3b25081a15b2028 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74077b61d6f54342bf9736b4a9f2eade in term 1.
I20240501 07:59:14.548529 9056 raft_consensus.cc:2454] T 0f9e2549fe294a93b3b25081a15b2028 P a505e7cb133c4392bfda8a4a20874ae9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74077b61d6f54342bf9736b4a9f2eade in term 1.
I20240501 07:59:14.548827 8728 leader_election.cc:302] T 0f9e2549fe294a93b3b25081a15b2028 P 74077b61d6f54342bf9736b4a9f2eade [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74077b61d6f54342bf9736b4a9f2eade, a505e7cb133c4392bfda8a4a20874ae9; no voters:
I20240501 07:59:14.549109 11723 raft_consensus.cc:2781] T 0f9e2549fe294a93b3b25081a15b2028 P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:14.549221 11723 raft_consensus.cc:686] T 0f9e2549fe294a93b3b25081a15b2028 P 74077b61d6f54342bf9736b4a9f2eade [term 1 LEADER]: Becoming Leader. State: Replica: 74077b61d6f54342bf9736b4a9f2eade, State: Running, Role: LEADER
I20240501 07:59:14.549353 11723 consensus_queue.cc:237] T 0f9e2549fe294a93b3b25081a15b2028 P 74077b61d6f54342bf9736b4a9f2eade [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } }
I20240501 07:59:14.550905 6575 consensus_queue.cc:237] T a4727883b2544ed58b356f41dc8017a9 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: true } }
I20240501 07:59:14.551137 6172 catalog_manager.cc:5484] T 0f9e2549fe294a93b3b25081a15b2028 P 74077b61d6f54342bf9736b4a9f2eade reported cstate change: term changed from 0 to 1, leader changed from <none> to 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133). New cstate: current_term: 1 leader_uuid: "74077b61d6f54342bf9736b4a9f2eade" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } health_report { overall_health: UNKNOWN } } }
I20240501 07:59:14.552376 6444 raft_consensus.cc:1261] T a4727883b2544ed58b356f41dc8017a9 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Refusing update from remote peer 3b164cbbc00f4e8eb3254ddd349a3ff9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:59:14.552769 11119 consensus_queue.cc:1035] T a4727883b2544ed58b356f41dc8017a9 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:14.552989 6708 raft_consensus.cc:1261] T a4727883b2544ed58b356f41dc8017a9 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Refusing update from remote peer 3b164cbbc00f4e8eb3254ddd349a3ff9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 07:59:14.553012 6512 consensus_peers.cc:485] T a4727883b2544ed58b356f41dc8017a9 P 3b164cbbc00f4e8eb3254ddd349a3ff9 -> Peer 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134:40141): Couldn't send request to peer 13d05ebdca8c4f489c597ce550dd773d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a4727883b2544ed58b356f41dc8017a9. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:14.553781 11119 consensus_queue.cc:1035] T a4727883b2544ed58b356f41dc8017a9 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:14.554726 11107 raft_consensus.cc:2932] T a4727883b2544ed58b356f41dc8017a9 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: true } } }
I20240501 07:59:14.555233 6444 raft_consensus.cc:2932] T a4727883b2544ed58b356f41dc8017a9 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: true } } }
I20240501 07:59:14.557157 6708 raft_consensus.cc:2932] T a4727883b2544ed58b356f41dc8017a9 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: true } } }
I20240501 07:59:14.557808 6172 catalog_manager.cc:5484] T a4727883b2544ed58b356f41dc8017a9 P 3b164cbbc00f4e8eb3254ddd349a3ff9 reported cstate change: config changed from index -1 to 4, NON_VOTER 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 07:59:14.573231 6575 raft_consensus.cc:1203] T 542c9ac08f7d4bec9b15a309c2e21364 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 07:59:14.593060 8924 raft_consensus.cc:1203] T 2e540f389b2a4aef93a020f549d11b67 P 13d05ebdca8c4f489c597ce550dd773d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240501 07:59:14.604604 8794 consensus_queue.cc:237] T 6e91683ff22243c1b3877733d74e2886 P 74077b61d6f54342bf9736b4a9f2eade [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } }
I20240501 07:59:14.606119 6707 raft_consensus.cc:1261] T 6e91683ff22243c1b3877733d74e2886 P 1f9cb72be98941b5a8e504cbe34132b9 [term 2 FOLLOWER]: Refusing update from remote peer 74077b61d6f54342bf9736b4a9f2eade: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 07:59:14.606499 11724 consensus_queue.cc:1035] T 6e91683ff22243c1b3877733d74e2886 P 74077b61d6f54342bf9736b4a9f2eade [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 07:59:14.606978 6313 raft_consensus.cc:1261] T 6e91683ff22243c1b3877733d74e2886 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 2 FOLLOWER]: Refusing update from remote peer 74077b61d6f54342bf9736b4a9f2eade: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 07:59:14.607326 11724 consensus_queue.cc:1035] T 6e91683ff22243c1b3877733d74e2886 P 74077b61d6f54342bf9736b4a9f2eade [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 07:59:14.608579 12123 raft_consensus.cc:2932] T 6e91683ff22243c1b3877733d74e2886 P 74077b61d6f54342bf9736b4a9f2eade [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } } }
I20240501 07:59:14.609176 6707 raft_consensus.cc:2932] T 6e91683ff22243c1b3877733d74e2886 P 1f9cb72be98941b5a8e504cbe34132b9 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } } }
I20240501 07:59:14.610177 6157 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6e91683ff22243c1b3877733d74e2886 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 07:59:14.611604 6172 catalog_manager.cc:5484] T 6e91683ff22243c1b3877733d74e2886 P 74077b61d6f54342bf9736b4a9f2eade reported cstate change: config changed from index 5 to 7, VOTER 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131) evicted. New cstate: current_term: 2 leader_uuid: "74077b61d6f54342bf9736b4a9f2eade" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 07:59:14.612033 6313 raft_consensus.cc:2932] T 6e91683ff22243c1b3877733d74e2886 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } } }
I20240501 07:59:14.614841 6556 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e91683ff22243c1b3877733d74e2886 with delete_type TABLET_DATA_TOMBSTONED (TS 3b164cbbc00f4e8eb3254ddd349a3ff9 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:53874
I20240501 07:59:14.615885 12354 tablet_replica.cc:330] T 6e91683ff22243c1b3877733d74e2886 P 3b164cbbc00f4e8eb3254ddd349a3ff9: stopping tablet replica
I20240501 07:59:14.616137 12354 raft_consensus.cc:2229] T 6e91683ff22243c1b3877733d74e2886 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:14.616353 12354 raft_consensus.cc:2258] T 6e91683ff22243c1b3877733d74e2886 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:14.617389 12354 ts_tablet_manager.cc:1875] T 6e91683ff22243c1b3877733d74e2886 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:59:14.619712 12354 ts_tablet_manager.cc:1888] T 6e91683ff22243c1b3877733d74e2886 P 3b164cbbc00f4e8eb3254ddd349a3ff9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240501 07:59:14.619779 12354 log.cc:1196] T 6e91683ff22243c1b3877733d74e2886 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/6e91683ff22243c1b3877733d74e2886
I20240501 07:59:14.620692 6160 catalog_manager.cc:4830] TS 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131:34471): tablet 6e91683ff22243c1b3877733d74e2886 (table rebalance_test_table_1 [id=d0e1f982440849beb494081bea9cb211]) successfully deleted
I20240501 07:59:14.644187 6172 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55950:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 07:59:14.651386 6707 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ff9dc384dbac47dc9795617c64688943"
dest_uuid: "1f9cb72be98941b5a8e504cbe34132b9"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:52710
I20240501 07:59:14.651541 6707 raft_consensus.cc:595] T ff9dc384dbac47dc9795617c64688943 P 1f9cb72be98941b5a8e504cbe34132b9 [term 2 LEADER]: Received request to transfer leadership
I20240501 07:59:14.651842 12237 tablet.cc:1722] T 1e06c2e0771541bdb61d1d995c93a543 P 8315d22c56f7439a9e1fc82f153f018c: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 07:59:14.652278 12238 tablet.cc:1722] T 1e06c2e0771541bdb61d1d995c93a543 P 13d05ebdca8c4f489c597ce550dd773d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 07:59:14.656106 12239 tablet.cc:1722] T 1e06c2e0771541bdb61d1d995c93a543 P a505e7cb133c4392bfda8a4a20874ae9: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 07:59:14.657474 10478 raft_consensus.cc:482] T bab84387a40d40c3b8af7f09eb462886 P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:14.657575 10478 raft_consensus.cc:504] T bab84387a40d40c3b8af7f09eb462886 P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } }
I20240501 07:59:14.657917 10478 leader_election.cc:288] T bab84387a40d40c3b8af7f09eb462886 P 13d05ebdca8c4f489c597ce550dd773d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130:32941), 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149)
I20240501 07:59:14.659147 9189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bab84387a40d40c3b8af7f09eb462886" candidate_uuid: "13d05ebdca8c4f489c597ce550dd773d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83f20e84af4c40d09a4804b6b26e697f" is_pre_election: true
I20240501 07:59:14.659272 9189 raft_consensus.cc:2454] T bab84387a40d40c3b8af7f09eb462886 P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 13d05ebdca8c4f489c597ce550dd773d in term 0.
I20240501 07:59:14.659610 6444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bab84387a40d40c3b8af7f09eb462886" candidate_uuid: "13d05ebdca8c4f489c597ce550dd773d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8315d22c56f7439a9e1fc82f153f018c" is_pre_election: true
I20240501 07:59:14.659672 8859 leader_election.cc:302] T bab84387a40d40c3b8af7f09eb462886 P 13d05ebdca8c4f489c597ce550dd773d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13d05ebdca8c4f489c597ce550dd773d, 83f20e84af4c40d09a4804b6b26e697f; no voters:
I20240501 07:59:14.659727 6444 raft_consensus.cc:2454] T bab84387a40d40c3b8af7f09eb462886 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 13d05ebdca8c4f489c597ce550dd773d in term 0.
I20240501 07:59:14.661540 11215 raft_consensus.cc:2781] T bab84387a40d40c3b8af7f09eb462886 P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:14.661610 11215 raft_consensus.cc:482] T bab84387a40d40c3b8af7f09eb462886 P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:14.661648 11215 raft_consensus.cc:3037] T bab84387a40d40c3b8af7f09eb462886 P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:14.662721 11215 raft_consensus.cc:504] T bab84387a40d40c3b8af7f09eb462886 P 13d05ebdca8c4f489c597ce550dd773d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } }
I20240501 07:59:14.663064 11215 leader_election.cc:288] T bab84387a40d40c3b8af7f09eb462886 P 13d05ebdca8c4f489c597ce550dd773d [CANDIDATE]: Term 1 election: Requested vote from peers 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130:32941), 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149)
I20240501 07:59:14.663512 6172 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 0bc706e85ce74cef97a4827ea4d4aa65: rebalancer_dynamic_table_new_name [id=b2142804a6254da283ecbc4af2be1c08] alter complete (version 3)
I20240501 07:59:14.664227 9189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bab84387a40d40c3b8af7f09eb462886" candidate_uuid: "13d05ebdca8c4f489c597ce550dd773d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83f20e84af4c40d09a4804b6b26e697f"
I20240501 07:59:14.664343 9189 raft_consensus.cc:3037] T bab84387a40d40c3b8af7f09eb462886 P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:14.665354 9189 raft_consensus.cc:2454] T bab84387a40d40c3b8af7f09eb462886 P 83f20e84af4c40d09a4804b6b26e697f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13d05ebdca8c4f489c597ce550dd773d in term 1.
I20240501 07:59:14.666079 8859 leader_election.cc:302] T bab84387a40d40c3b8af7f09eb462886 P 13d05ebdca8c4f489c597ce550dd773d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13d05ebdca8c4f489c597ce550dd773d, 83f20e84af4c40d09a4804b6b26e697f; no voters:
I20240501 07:59:14.666350 12019 raft_consensus.cc:2781] T bab84387a40d40c3b8af7f09eb462886 P 13d05ebdca8c4f489c597ce550dd773d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:14.666436 12019 raft_consensus.cc:686] T bab84387a40d40c3b8af7f09eb462886 P 13d05ebdca8c4f489c597ce550dd773d [term 1 LEADER]: Becoming Leader. State: Replica: 13d05ebdca8c4f489c597ce550dd773d, State: Running, Role: LEADER
I20240501 07:59:14.666550 12019 consensus_queue.cc:237] T bab84387a40d40c3b8af7f09eb462886 P 13d05ebdca8c4f489c597ce550dd773d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } }
I20240501 07:59:14.667601 6444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bab84387a40d40c3b8af7f09eb462886" candidate_uuid: "13d05ebdca8c4f489c597ce550dd773d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8315d22c56f7439a9e1fc82f153f018c"
I20240501 07:59:14.667726 6444 raft_consensus.cc:3037] T bab84387a40d40c3b8af7f09eb462886 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:14.668592 6170 catalog_manager.cc:5484] T bab84387a40d40c3b8af7f09eb462886 P 13d05ebdca8c4f489c597ce550dd773d reported cstate change: term changed from 0 to 1, leader changed from <none> to 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134). New cstate: current_term: 1 leader_uuid: "13d05ebdca8c4f489c597ce550dd773d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } health_report { overall_health: UNKNOWN } } }
I20240501 07:59:14.668752 6444 raft_consensus.cc:2454] T bab84387a40d40c3b8af7f09eb462886 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13d05ebdca8c4f489c597ce550dd773d in term 1.
I20240501 07:59:14.683451 12391 ts_tablet_manager.cc:902] T 114b6dbafa1c4c44ac42f1d2293e3125 P 74077b61d6f54342bf9736b4a9f2eade: Initiating tablet copy from peer 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131:34471)
I20240501 07:59:14.684103 12391 tablet_copy_client.cc:308] T 114b6dbafa1c4c44ac42f1d2293e3125 P 74077b61d6f54342bf9736b4a9f2eade: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:34471
I20240501 07:59:14.686486 6597 tablet_copy_service.cc:140] P 3b164cbbc00f4e8eb3254ddd349a3ff9: Received BeginTabletCopySession request for tablet 114b6dbafa1c4c44ac42f1d2293e3125 from peer 74077b61d6f54342bf9736b4a9f2eade ({username='slave'} at 127.0.99.133:57559)
I20240501 07:59:14.686609 6597 tablet_copy_service.cc:161] P 3b164cbbc00f4e8eb3254ddd349a3ff9: Beginning new tablet copy session on tablet 114b6dbafa1c4c44ac42f1d2293e3125 from peer 74077b61d6f54342bf9736b4a9f2eade at {username='slave'} at 127.0.99.133:57559: session id = 74077b61d6f54342bf9736b4a9f2eade-114b6dbafa1c4c44ac42f1d2293e3125
I20240501 07:59:14.687515 6597 tablet_copy_source_session.cc:205] T 114b6dbafa1c4c44ac42f1d2293e3125 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:14.688367 12391 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 114b6dbafa1c4c44ac42f1d2293e3125. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:14.688532 8925 consensus_queue.cc:237] T 59e171892fc1414fbcfd6a8c090854f3 P 13d05ebdca8c4f489c597ce550dd773d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { replace: true } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: true } }
W20240501 07:59:14.690151 8861 consensus_peers.cc:485] T 59e171892fc1414fbcfd6a8c090854f3 P 13d05ebdca8c4f489c597ce550dd773d -> Peer 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131:34471): Couldn't send request to peer 3b164cbbc00f4e8eb3254ddd349a3ff9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 59e171892fc1414fbcfd6a8c090854f3. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:14.690168 8794 raft_consensus.cc:1261] T 59e171892fc1414fbcfd6a8c090854f3 P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Refusing update from remote peer 13d05ebdca8c4f489c597ce550dd773d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:14.690193 9189 raft_consensus.cc:1261] T 59e171892fc1414fbcfd6a8c090854f3 P 83f20e84af4c40d09a4804b6b26e697f [term 1 FOLLOWER]: Refusing update from remote peer 13d05ebdca8c4f489c597ce550dd773d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:14.690565 12019 consensus_queue.cc:1035] T 59e171892fc1414fbcfd6a8c090854f3 P 13d05ebdca8c4f489c597ce550dd773d [LEADER]: Connected to new peer: Peer: permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 07:59:14.690697 12391 tablet_copy_client.cc:783] T 114b6dbafa1c4c44ac42f1d2293e3125 P 74077b61d6f54342bf9736b4a9f2eade: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:14.690759 11215 consensus_queue.cc:1035] T 59e171892fc1414fbcfd6a8c090854f3 P 13d05ebdca8c4f489c597ce550dd773d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 07:59:14.690872 12391 tablet_copy_client.cc:647] T 114b6dbafa1c4c44ac42f1d2293e3125 P 74077b61d6f54342bf9736b4a9f2eade: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:14.692783 12391 tablet_copy_client.cc:523] T 114b6dbafa1c4c44ac42f1d2293e3125 P 74077b61d6f54342bf9736b4a9f2eade: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:14.693114 11215 raft_consensus.cc:2932] T 59e171892fc1414fbcfd6a8c090854f3 P 13d05ebdca8c4f489c597ce550dd773d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { replace: true } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: true } } }
I20240501 07:59:14.694092 12391 tablet_bootstrap.cc:492] T 114b6dbafa1c4c44ac42f1d2293e3125 P 74077b61d6f54342bf9736b4a9f2eade: Bootstrap starting.
I20240501 07:59:14.695250 12123 raft_consensus.cc:482] T 36aaf192042f4320b2526322424dc640 P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:14.695326 12123 raft_consensus.cc:504] T 36aaf192042f4320b2526322424dc640 P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } }
I20240501 07:59:14.695647 12123 leader_election.cc:288] T 36aaf192042f4320b2526322424dc640 P 74077b61d6f54342bf9736b4a9f2eade [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149), 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134:40141)
I20240501 07:59:14.696173 8925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36aaf192042f4320b2526322424dc640" candidate_uuid: "74077b61d6f54342bf9736b4a9f2eade" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13d05ebdca8c4f489c597ce550dd773d" is_pre_election: true
I20240501 07:59:14.696300 8925 raft_consensus.cc:2454] T 36aaf192042f4320b2526322424dc640 P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74077b61d6f54342bf9736b4a9f2eade in term 0.
I20240501 07:59:14.696271 9189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36aaf192042f4320b2526322424dc640" candidate_uuid: "74077b61d6f54342bf9736b4a9f2eade" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83f20e84af4c40d09a4804b6b26e697f" is_pre_election: true
I20240501 07:59:14.696399 9189 raft_consensus.cc:2454] T 36aaf192042f4320b2526322424dc640 P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74077b61d6f54342bf9736b4a9f2eade in term 0.
I20240501 07:59:14.696388 9188 raft_consensus.cc:2932] T 59e171892fc1414fbcfd6a8c090854f3 P 83f20e84af4c40d09a4804b6b26e697f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { replace: true } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: true } } }
I20240501 07:59:14.696817 8728 leader_election.cc:302] T 36aaf192042f4320b2526322424dc640 P 74077b61d6f54342bf9736b4a9f2eade [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74077b61d6f54342bf9736b4a9f2eade, 83f20e84af4c40d09a4804b6b26e697f; no voters:
I20240501 07:59:14.697062 12123 raft_consensus.cc:2781] T 36aaf192042f4320b2526322424dc640 P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:14.697113 12123 raft_consensus.cc:482] T 36aaf192042f4320b2526322424dc640 P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:14.697154 12123 raft_consensus.cc:3037] T 36aaf192042f4320b2526322424dc640 P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:14.698022 12123 raft_consensus.cc:504] T 36aaf192042f4320b2526322424dc640 P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } }
I20240501 07:59:14.698354 12123 leader_election.cc:288] T 36aaf192042f4320b2526322424dc640 P 74077b61d6f54342bf9736b4a9f2eade [CANDIDATE]: Term 1 election: Requested vote from peers 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149), 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134:40141)
I20240501 07:59:14.698801 6170 catalog_manager.cc:5484] T 59e171892fc1414fbcfd6a8c090854f3 P 13d05ebdca8c4f489c597ce550dd773d reported cstate change: config changed from index -1 to 2, NON_VOTER 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131) added. New cstate: current_term: 1 leader_uuid: "13d05ebdca8c4f489c597ce550dd773d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 07:59:14.700706 8925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36aaf192042f4320b2526322424dc640" candidate_uuid: "74077b61d6f54342bf9736b4a9f2eade" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13d05ebdca8c4f489c597ce550dd773d"
I20240501 07:59:14.700824 8925 raft_consensus.cc:3037] T 36aaf192042f4320b2526322424dc640 P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:14.701915 8925 raft_consensus.cc:2454] T 36aaf192042f4320b2526322424dc640 P 13d05ebdca8c4f489c597ce550dd773d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74077b61d6f54342bf9736b4a9f2eade in term 1.
I20240501 07:59:14.702044 9188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36aaf192042f4320b2526322424dc640" candidate_uuid: "74077b61d6f54342bf9736b4a9f2eade" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83f20e84af4c40d09a4804b6b26e697f"
I20240501 07:59:14.702179 9188 raft_consensus.cc:3037] T 36aaf192042f4320b2526322424dc640 P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:14.703212 9188 raft_consensus.cc:2454] T 36aaf192042f4320b2526322424dc640 P 83f20e84af4c40d09a4804b6b26e697f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74077b61d6f54342bf9736b4a9f2eade in term 1.
I20240501 07:59:14.703601 8729 leader_election.cc:302] T 36aaf192042f4320b2526322424dc640 P 74077b61d6f54342bf9736b4a9f2eade [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13d05ebdca8c4f489c597ce550dd773d, 74077b61d6f54342bf9736b4a9f2eade; no voters:
I20240501 07:59:14.703812 12123 raft_consensus.cc:2781] T 36aaf192042f4320b2526322424dc640 P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:14.703893 12123 raft_consensus.cc:686] T 36aaf192042f4320b2526322424dc640 P 74077b61d6f54342bf9736b4a9f2eade [term 1 LEADER]: Becoming Leader. State: Replica: 74077b61d6f54342bf9736b4a9f2eade, State: Running, Role: LEADER
I20240501 07:59:14.704015 12123 consensus_queue.cc:237] T 36aaf192042f4320b2526322424dc640 P 74077b61d6f54342bf9736b4a9f2eade [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } }
I20240501 07:59:14.706104 6171 catalog_manager.cc:5484] T 36aaf192042f4320b2526322424dc640 P 74077b61d6f54342bf9736b4a9f2eade reported cstate change: term changed from 0 to 1, leader changed from <none> to 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133). New cstate: current_term: 1 leader_uuid: "74077b61d6f54342bf9736b4a9f2eade" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } health_report { overall_health: UNKNOWN } } }
I20240501 07:59:14.708758 12049 raft_consensus.cc:979] T ff9dc384dbac47dc9795617c64688943 P 1f9cb72be98941b5a8e504cbe34132b9: : Instructing follower 13d05ebdca8c4f489c597ce550dd773d to start an election
I20240501 07:59:14.708858 12049 raft_consensus.cc:1067] T ff9dc384dbac47dc9795617c64688943 P 1f9cb72be98941b5a8e504cbe34132b9 [term 2 LEADER]: Signalling peer 13d05ebdca8c4f489c597ce550dd773d to start an election
I20240501 07:59:14.709362 8793 raft_consensus.cc:2932] T 59e171892fc1414fbcfd6a8c090854f3 P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { replace: true } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: true } } }
I20240501 07:59:14.711839 8925 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ff9dc384dbac47dc9795617c64688943"
dest_uuid: "13d05ebdca8c4f489c597ce550dd773d"
from {username='slave'} at 127.0.99.132:43109
I20240501 07:59:14.711982 8925 raft_consensus.cc:482] T ff9dc384dbac47dc9795617c64688943 P 13d05ebdca8c4f489c597ce550dd773d [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:59:14.712039 8925 raft_consensus.cc:3037] T ff9dc384dbac47dc9795617c64688943 P 13d05ebdca8c4f489c597ce550dd773d [term 2 FOLLOWER]: Advancing to term 3
I20240501 07:59:14.713086 8925 raft_consensus.cc:504] T ff9dc384dbac47dc9795617c64688943 P 13d05ebdca8c4f489c597ce550dd773d [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { replace: true } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false } }
I20240501 07:59:14.713536 8925 leader_election.cc:288] T ff9dc384dbac47dc9795617c64688943 P 13d05ebdca8c4f489c597ce550dd773d [CANDIDATE]: Term 3 election: Requested vote from peers 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129:46875), 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132:36773), 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149)
I20240501 07:59:14.714011 6707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9dc384dbac47dc9795617c64688943" candidate_uuid: "13d05ebdca8c4f489c597ce550dd773d" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: true dest_uuid: "1f9cb72be98941b5a8e504cbe34132b9"
I20240501 07:59:14.714155 6707 raft_consensus.cc:3032] T ff9dc384dbac47dc9795617c64688943 P 1f9cb72be98941b5a8e504cbe34132b9 [term 2 LEADER]: Stepping down as leader of term 2
I20240501 07:59:14.714203 6707 raft_consensus.cc:729] T ff9dc384dbac47dc9795617c64688943 P 1f9cb72be98941b5a8e504cbe34132b9 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1f9cb72be98941b5a8e504cbe34132b9, State: Running, Role: LEADER
I20240501 07:59:14.714375 6707 consensus_queue.cc:260] T ff9dc384dbac47dc9795617c64688943 P 1f9cb72be98941b5a8e504cbe34132b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 9, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { replace: true } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false } }
I20240501 07:59:14.714603 6707 raft_consensus.cc:3037] T ff9dc384dbac47dc9795617c64688943 P 1f9cb72be98941b5a8e504cbe34132b9 [term 2 FOLLOWER]: Advancing to term 3
I20240501 07:59:14.714648 9189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9dc384dbac47dc9795617c64688943" candidate_uuid: "13d05ebdca8c4f489c597ce550dd773d" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: true dest_uuid: "83f20e84af4c40d09a4804b6b26e697f"
I20240501 07:59:14.714762 9189 raft_consensus.cc:3037] T ff9dc384dbac47dc9795617c64688943 P 83f20e84af4c40d09a4804b6b26e697f [term 2 FOLLOWER]: Advancing to term 3
I20240501 07:59:14.715646 6707 raft_consensus.cc:2454] T ff9dc384dbac47dc9795617c64688943 P 1f9cb72be98941b5a8e504cbe34132b9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13d05ebdca8c4f489c597ce550dd773d in term 3.
I20240501 07:59:14.715777 9189 raft_consensus.cc:2454] T ff9dc384dbac47dc9795617c64688943 P 83f20e84af4c40d09a4804b6b26e697f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13d05ebdca8c4f489c597ce550dd773d in term 3.
I20240501 07:59:14.716384 8859 leader_election.cc:302] T ff9dc384dbac47dc9795617c64688943 P 13d05ebdca8c4f489c597ce550dd773d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 13d05ebdca8c4f489c597ce550dd773d, 1f9cb72be98941b5a8e504cbe34132b9, 83f20e84af4c40d09a4804b6b26e697f; no voters:
I20240501 07:59:14.716565 12019 raft_consensus.cc:2781] T ff9dc384dbac47dc9795617c64688943 P 13d05ebdca8c4f489c597ce550dd773d [term 3 FOLLOWER]: Leader election won for term 3
I20240501 07:59:14.716672 12019 raft_consensus.cc:686] T ff9dc384dbac47dc9795617c64688943 P 13d05ebdca8c4f489c597ce550dd773d [term 3 LEADER]: Becoming Leader. State: Replica: 13d05ebdca8c4f489c597ce550dd773d, State: Running, Role: LEADER
I20240501 07:59:14.716842 12019 consensus_queue.cc:237] T ff9dc384dbac47dc9795617c64688943 P 13d05ebdca8c4f489c597ce550dd773d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 9, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { replace: true } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false } }
I20240501 07:59:14.717190 6313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9dc384dbac47dc9795617c64688943" candidate_uuid: "13d05ebdca8c4f489c597ce550dd773d" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: true dest_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f"
I20240501 07:59:14.717321 6313 raft_consensus.cc:3037] T ff9dc384dbac47dc9795617c64688943 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 2 FOLLOWER]: Advancing to term 3
I20240501 07:59:14.718415 6313 raft_consensus.cc:2454] T ff9dc384dbac47dc9795617c64688943 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13d05ebdca8c4f489c597ce550dd773d in term 3.
I20240501 07:59:14.719405 6171 catalog_manager.cc:5484] T ff9dc384dbac47dc9795617c64688943 P 13d05ebdca8c4f489c597ce550dd773d reported cstate change: term changed from 2 to 3, leader changed from 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132) to 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134). New cstate: current_term: 3 leader_uuid: "13d05ebdca8c4f489c597ce550dd773d" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 07:59:14.724738 12391 tablet_bootstrap.cc:492] T 114b6dbafa1c4c44ac42f1d2293e3125 P 74077b61d6f54342bf9736b4a9f2eade: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:14.725102 12391 tablet_bootstrap.cc:492] T 114b6dbafa1c4c44ac42f1d2293e3125 P 74077b61d6f54342bf9736b4a9f2eade: Bootstrap complete.
I20240501 07:59:14.725344 12391 ts_tablet_manager.cc:1367] T 114b6dbafa1c4c44ac42f1d2293e3125 P 74077b61d6f54342bf9736b4a9f2eade: Time spent bootstrapping tablet: real 0.031s user 0.026s sys 0.003s
I20240501 07:59:14.725776 12391 raft_consensus.cc:348] T 114b6dbafa1c4c44ac42f1d2293e3125 P 74077b61d6f54342bf9736b4a9f2eade [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: true } }
I20240501 07:59:14.726006 12391 raft_consensus.cc:729] T 114b6dbafa1c4c44ac42f1d2293e3125 P 74077b61d6f54342bf9736b4a9f2eade [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 74077b61d6f54342bf9736b4a9f2eade, State: Initialized, Role: LEARNER
I20240501 07:59:14.726176 12391 consensus_queue.cc:260] T 114b6dbafa1c4c44ac42f1d2293e3125 P 74077b61d6f54342bf9736b4a9f2eade [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: true } }
I20240501 07:59:14.726619 12391 ts_tablet_manager.cc:1398] T 114b6dbafa1c4c44ac42f1d2293e3125 P 74077b61d6f54342bf9736b4a9f2eade: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:14.727183 6597 tablet_copy_service.cc:342] P 3b164cbbc00f4e8eb3254ddd349a3ff9: Request end of tablet copy session 74077b61d6f54342bf9736b4a9f2eade-114b6dbafa1c4c44ac42f1d2293e3125 received from {username='slave'} at 127.0.99.133:57559
I20240501 07:59:14.727257 6597 tablet_copy_service.cc:433] P 3b164cbbc00f4e8eb3254ddd349a3ff9: ending tablet copy session 74077b61d6f54342bf9736b4a9f2eade-114b6dbafa1c4c44ac42f1d2293e3125 on tablet 114b6dbafa1c4c44ac42f1d2293e3125 with peer 74077b61d6f54342bf9736b4a9f2eade
I20240501 07:59:14.768328 6171 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55950:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 07:59:14.768518 6171 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55950:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 07:59:14.770700 6171 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 0bc706e85ce74cef97a4827ea4d4aa65: Sending DeleteTablet for 3 replicas of tablet 1e06c2e0771541bdb61d1d995c93a543
I20240501 07:59:14.771288 8904 tablet_service.cc:1514] Processing DeleteTablet for tablet 1e06c2e0771541bdb61d1d995c93a543 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:14 UTC) from {username='slave'} at 127.0.0.1:47202
I20240501 07:59:14.771528 9035 tablet_service.cc:1514] Processing DeleteTablet for tablet 1e06c2e0771541bdb61d1d995c93a543 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:14 UTC) from {username='slave'} at 127.0.0.1:44236
I20240501 07:59:14.771602 12446 tablet_replica.cc:330] T 1e06c2e0771541bdb61d1d995c93a543 P 13d05ebdca8c4f489c597ce550dd773d: stopping tablet replica
I20240501 07:59:14.771589 6424 tablet_service.cc:1514] Processing DeleteTablet for tablet 1e06c2e0771541bdb61d1d995c93a543 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:14 UTC) from {username='slave'} at 127.0.0.1:45034
I20240501 07:59:14.771755 12446 raft_consensus.cc:2229] T 1e06c2e0771541bdb61d1d995c93a543 P 13d05ebdca8c4f489c597ce550dd773d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:14.771780 12057 tablet_replica.cc:330] T 1e06c2e0771541bdb61d1d995c93a543 P 8315d22c56f7439a9e1fc82f153f018c: stopping tablet replica
I20240501 07:59:14.771883 12446 raft_consensus.cc:2258] T 1e06c2e0771541bdb61d1d995c93a543 P 13d05ebdca8c4f489c597ce550dd773d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:14.771919 12057 raft_consensus.cc:2229] T 1e06c2e0771541bdb61d1d995c93a543 P 8315d22c56f7439a9e1fc82f153f018c [term 1 LEADER]: Raft consensus shutting down.
I20240501 07:59:14.772130 12057 raft_consensus.cc:2258] T 1e06c2e0771541bdb61d1d995c93a543 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:14.772785 12057 ts_tablet_manager.cc:1875] T 1e06c2e0771541bdb61d1d995c93a543 P 8315d22c56f7439a9e1fc82f153f018c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:59:14.773505 12446 ts_tablet_manager.cc:1875] T 1e06c2e0771541bdb61d1d995c93a543 P 13d05ebdca8c4f489c597ce550dd773d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:59:14.774104 12447 tablet_replica.cc:330] T 1e06c2e0771541bdb61d1d995c93a543 P a505e7cb133c4392bfda8a4a20874ae9: stopping tablet replica
I20240501 07:59:14.774271 12447 raft_consensus.cc:2229] T 1e06c2e0771541bdb61d1d995c93a543 P a505e7cb133c4392bfda8a4a20874ae9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:14.774430 12447 raft_consensus.cc:2258] T 1e06c2e0771541bdb61d1d995c93a543 P a505e7cb133c4392bfda8a4a20874ae9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:14.775157 12447 ts_tablet_manager.cc:1875] T 1e06c2e0771541bdb61d1d995c93a543 P a505e7cb133c4392bfda8a4a20874ae9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:59:14.775338 12057 ts_tablet_manager.cc:1888] T 1e06c2e0771541bdb61d1d995c93a543 P 8315d22c56f7439a9e1fc82f153f018c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:59:14.775409 12057 log.cc:1196] T 1e06c2e0771541bdb61d1d995c93a543 P 8315d22c56f7439a9e1fc82f153f018c: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/1e06c2e0771541bdb61d1d995c93a543
I20240501 07:59:14.775673 12057 ts_tablet_manager.cc:1909] T 1e06c2e0771541bdb61d1d995c93a543 P 8315d22c56f7439a9e1fc82f153f018c: Deleting consensus metadata
I20240501 07:59:14.776417 6159 catalog_manager.cc:4830] TS 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130:32941): tablet 1e06c2e0771541bdb61d1d995c93a543 (table rebalancer_dynamic_table_new_name [id=b2142804a6254da283ecbc4af2be1c08]) successfully deleted
I20240501 07:59:14.776677 12446 ts_tablet_manager.cc:1888] T 1e06c2e0771541bdb61d1d995c93a543 P 13d05ebdca8c4f489c597ce550dd773d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:59:14.776772 12446 log.cc:1196] T 1e06c2e0771541bdb61d1d995c93a543 P 13d05ebdca8c4f489c597ce550dd773d: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/1e06c2e0771541bdb61d1d995c93a543
I20240501 07:59:14.777065 12446 ts_tablet_manager.cc:1909] T 1e06c2e0771541bdb61d1d995c93a543 P 13d05ebdca8c4f489c597ce550dd773d: Deleting consensus metadata
I20240501 07:59:14.777868 6159 catalog_manager.cc:4830] TS 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134:40141): tablet 1e06c2e0771541bdb61d1d995c93a543 (table rebalancer_dynamic_table_new_name [id=b2142804a6254da283ecbc4af2be1c08]) successfully deleted
I20240501 07:59:14.779750 12447 ts_tablet_manager.cc:1888] T 1e06c2e0771541bdb61d1d995c93a543 P a505e7cb133c4392bfda8a4a20874ae9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:59:14.779835 12447 log.cc:1196] T 1e06c2e0771541bdb61d1d995c93a543 P a505e7cb133c4392bfda8a4a20874ae9: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/1e06c2e0771541bdb61d1d995c93a543
I20240501 07:59:14.780138 12447 ts_tablet_manager.cc:1909] T 1e06c2e0771541bdb61d1d995c93a543 P a505e7cb133c4392bfda8a4a20874ae9: Deleting consensus metadata
I20240501 07:59:14.780822 6158 catalog_manager.cc:4830] TS a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): tablet 1e06c2e0771541bdb61d1d995c93a543 (table rebalancer_dynamic_table_new_name [id=b2142804a6254da283ecbc4af2be1c08]) successfully deleted
I20240501 07:59:14.853130 12468 ts_tablet_manager.cc:902] T 6c3c0d6fe41f4ae196ae29beb9b68125 P 13d05ebdca8c4f489c597ce550dd773d: Initiating tablet copy from peer 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129:46875)
I20240501 07:59:14.853657 12468 tablet_copy_client.cc:308] T 6c3c0d6fe41f4ae196ae29beb9b68125 P 13d05ebdca8c4f489c597ce550dd773d: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:46875
I20240501 07:59:14.854189 6335 tablet_copy_service.cc:140] P 8b8b3ad3aa8f4053b2167dd550f2086f: Received BeginTabletCopySession request for tablet 6c3c0d6fe41f4ae196ae29beb9b68125 from peer 13d05ebdca8c4f489c597ce550dd773d ({username='slave'} at 127.0.99.134:55617)
I20240501 07:59:14.854305 6335 tablet_copy_service.cc:161] P 8b8b3ad3aa8f4053b2167dd550f2086f: Beginning new tablet copy session on tablet 6c3c0d6fe41f4ae196ae29beb9b68125 from peer 13d05ebdca8c4f489c597ce550dd773d at {username='slave'} at 127.0.99.134:55617: session id = 13d05ebdca8c4f489c597ce550dd773d-6c3c0d6fe41f4ae196ae29beb9b68125
I20240501 07:59:14.855262 6335 tablet_copy_source_session.cc:205] T 6c3c0d6fe41f4ae196ae29beb9b68125 P 8b8b3ad3aa8f4053b2167dd550f2086f: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:14.856149 12468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c3c0d6fe41f4ae196ae29beb9b68125. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:14.857949 12473 ts_tablet_manager.cc:902] T 2f9e82c3f1df4d1292e37e9cea07186e P 8315d22c56f7439a9e1fc82f153f018c: Initiating tablet copy from peer 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133:39859)
I20240501 07:59:14.858505 12473 tablet_copy_client.cc:308] T 2f9e82c3f1df4d1292e37e9cea07186e P 8315d22c56f7439a9e1fc82f153f018c: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.133:39859
I20240501 07:59:14.858665 12468 tablet_copy_client.cc:783] T 6c3c0d6fe41f4ae196ae29beb9b68125 P 13d05ebdca8c4f489c597ce550dd773d: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:14.858824 12468 tablet_copy_client.cc:647] T 6c3c0d6fe41f4ae196ae29beb9b68125 P 13d05ebdca8c4f489c597ce550dd773d: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:14.859019 8814 tablet_copy_service.cc:140] P 74077b61d6f54342bf9736b4a9f2eade: Received BeginTabletCopySession request for tablet 2f9e82c3f1df4d1292e37e9cea07186e from peer 8315d22c56f7439a9e1fc82f153f018c ({username='slave'} at 127.0.99.130:51123)
I20240501 07:59:14.859115 8814 tablet_copy_service.cc:161] P 74077b61d6f54342bf9736b4a9f2eade: Beginning new tablet copy session on tablet 2f9e82c3f1df4d1292e37e9cea07186e from peer 8315d22c56f7439a9e1fc82f153f018c at {username='slave'} at 127.0.99.130:51123: session id = 8315d22c56f7439a9e1fc82f153f018c-2f9e82c3f1df4d1292e37e9cea07186e
I20240501 07:59:14.859946 12468 tablet_copy_client.cc:523] T 6c3c0d6fe41f4ae196ae29beb9b68125 P 13d05ebdca8c4f489c597ce550dd773d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:14.860128 8814 tablet_copy_source_session.cc:205] T 2f9e82c3f1df4d1292e37e9cea07186e P 74077b61d6f54342bf9736b4a9f2eade: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:14.861038 12473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f9e82c3f1df4d1292e37e9cea07186e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:14.861191 12468 tablet_bootstrap.cc:492] T 6c3c0d6fe41f4ae196ae29beb9b68125 P 13d05ebdca8c4f489c597ce550dd773d: Bootstrap starting.
I20240501 07:59:14.863155 12473 tablet_copy_client.cc:783] T 2f9e82c3f1df4d1292e37e9cea07186e P 8315d22c56f7439a9e1fc82f153f018c: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:14.863332 12473 tablet_copy_client.cc:647] T 2f9e82c3f1df4d1292e37e9cea07186e P 8315d22c56f7439a9e1fc82f153f018c: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:14.864488 12473 tablet_copy_client.cc:523] T 2f9e82c3f1df4d1292e37e9cea07186e P 8315d22c56f7439a9e1fc82f153f018c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:14.865753 12473 tablet_bootstrap.cc:492] T 2f9e82c3f1df4d1292e37e9cea07186e P 8315d22c56f7439a9e1fc82f153f018c: Bootstrap starting.
I20240501 07:59:14.874194 6171 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55950:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 07:59:14.875198 12473 tablet_bootstrap.cc:492] T 2f9e82c3f1df4d1292e37e9cea07186e P 8315d22c56f7439a9e1fc82f153f018c: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:14.876832 12473 tablet_bootstrap.cc:492] T 2f9e82c3f1df4d1292e37e9cea07186e P 8315d22c56f7439a9e1fc82f153f018c: Bootstrap complete.
I20240501 07:59:14.877439 12473 ts_tablet_manager.cc:1367] T 2f9e82c3f1df4d1292e37e9cea07186e P 8315d22c56f7439a9e1fc82f153f018c: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.002s
I20240501 07:59:14.878062 12473 raft_consensus.cc:348] T 2f9e82c3f1df4d1292e37e9cea07186e P 8315d22c56f7439a9e1fc82f153f018c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { replace: true } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { promote: true } }
I20240501 07:59:14.878335 12473 raft_consensus.cc:729] T 2f9e82c3f1df4d1292e37e9cea07186e P 8315d22c56f7439a9e1fc82f153f018c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8315d22c56f7439a9e1fc82f153f018c, State: Initialized, Role: LEARNER
I20240501 07:59:14.878450 12473 consensus_queue.cc:260] T 2f9e82c3f1df4d1292e37e9cea07186e P 8315d22c56f7439a9e1fc82f153f018c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { replace: true } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { promote: true } }
I20240501 07:59:14.879230 12473 ts_tablet_manager.cc:1398] T 2f9e82c3f1df4d1292e37e9cea07186e P 8315d22c56f7439a9e1fc82f153f018c: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 07:59:14.879724 8814 tablet_copy_service.cc:342] P 74077b61d6f54342bf9736b4a9f2eade: Request end of tablet copy session 8315d22c56f7439a9e1fc82f153f018c-2f9e82c3f1df4d1292e37e9cea07186e received from {username='slave'} at 127.0.99.130:51123
I20240501 07:59:14.879818 8814 tablet_copy_service.cc:433] P 74077b61d6f54342bf9736b4a9f2eade: ending tablet copy session 8315d22c56f7439a9e1fc82f153f018c-2f9e82c3f1df4d1292e37e9cea07186e on tablet 2f9e82c3f1df4d1292e37e9cea07186e with peer 8315d22c56f7439a9e1fc82f153f018c
I20240501 07:59:14.880507 8774 tablet_service.cc:1467] Processing CreateTablet for tablet f2ce44c1d8c44b9789b21af5e889a419 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=956c08c49bed4f7ea842f14bdfec9e03]), partition=
I20240501 07:59:14.880662 6556 tablet_service.cc:1467] Processing CreateTablet for tablet f2ce44c1d8c44b9789b21af5e889a419 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=956c08c49bed4f7ea842f14bdfec9e03]), partition=
I20240501 07:59:14.880769 8774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2ce44c1d8c44b9789b21af5e889a419. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:14.880789 9169 tablet_service.cc:1467] Processing CreateTablet for tablet f2ce44c1d8c44b9789b21af5e889a419 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=956c08c49bed4f7ea842f14bdfec9e03]), partition=
I20240501 07:59:14.880879 6556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2ce44c1d8c44b9789b21af5e889a419. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:14.881072 9169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2ce44c1d8c44b9789b21af5e889a419. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:14.883281 12284 tablet_bootstrap.cc:492] T f2ce44c1d8c44b9789b21af5e889a419 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Bootstrap starting.
I20240501 07:59:14.883296 12283 tablet_bootstrap.cc:492] T f2ce44c1d8c44b9789b21af5e889a419 P 74077b61d6f54342bf9736b4a9f2eade: Bootstrap starting.
I20240501 07:59:14.883987 12468 tablet_bootstrap.cc:492] T 6c3c0d6fe41f4ae196ae29beb9b68125 P 13d05ebdca8c4f489c597ce550dd773d: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:14.884093 12283 tablet_bootstrap.cc:654] T f2ce44c1d8c44b9789b21af5e889a419 P 74077b61d6f54342bf9736b4a9f2eade: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:14.884172 12284 tablet_bootstrap.cc:654] T f2ce44c1d8c44b9789b21af5e889a419 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:14.884248 12468 tablet_bootstrap.cc:492] T 6c3c0d6fe41f4ae196ae29beb9b68125 P 13d05ebdca8c4f489c597ce550dd773d: Bootstrap complete.
I20240501 07:59:14.884467 12468 ts_tablet_manager.cc:1367] T 6c3c0d6fe41f4ae196ae29beb9b68125 P 13d05ebdca8c4f489c597ce550dd773d: Time spent bootstrapping tablet: real 0.023s user 0.020s sys 0.001s
I20240501 07:59:14.884781 12280 tablet_bootstrap.cc:492] T f2ce44c1d8c44b9789b21af5e889a419 P 83f20e84af4c40d09a4804b6b26e697f: Bootstrap starting.
I20240501 07:59:14.884913 12468 raft_consensus.cc:348] T 6c3c0d6fe41f4ae196ae29beb9b68125 P 13d05ebdca8c4f489c597ce550dd773d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: true } }
I20240501 07:59:14.885078 12468 raft_consensus.cc:729] T 6c3c0d6fe41f4ae196ae29beb9b68125 P 13d05ebdca8c4f489c597ce550dd773d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 13d05ebdca8c4f489c597ce550dd773d, State: Initialized, Role: LEARNER
I20240501 07:59:14.885185 12468 consensus_queue.cc:260] T 6c3c0d6fe41f4ae196ae29beb9b68125 P 13d05ebdca8c4f489c597ce550dd773d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: true } }
I20240501 07:59:14.885488 12468 ts_tablet_manager.cc:1398] T 6c3c0d6fe41f4ae196ae29beb9b68125 P 13d05ebdca8c4f489c597ce550dd773d: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240501 07:59:14.885761 12280 tablet_bootstrap.cc:654] T f2ce44c1d8c44b9789b21af5e889a419 P 83f20e84af4c40d09a4804b6b26e697f: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:14.886746 6335 tablet_copy_service.cc:342] P 8b8b3ad3aa8f4053b2167dd550f2086f: Request end of tablet copy session 13d05ebdca8c4f489c597ce550dd773d-6c3c0d6fe41f4ae196ae29beb9b68125 received from {username='slave'} at 127.0.99.134:55617
I20240501 07:59:14.886834 6335 tablet_copy_service.cc:433] P 8b8b3ad3aa8f4053b2167dd550f2086f: ending tablet copy session 13d05ebdca8c4f489c597ce550dd773d-6c3c0d6fe41f4ae196ae29beb9b68125 on tablet 6c3c0d6fe41f4ae196ae29beb9b68125 with peer 13d05ebdca8c4f489c597ce550dd773d
I20240501 07:59:14.888492 12284 tablet_bootstrap.cc:492] T f2ce44c1d8c44b9789b21af5e889a419 P 3b164cbbc00f4e8eb3254ddd349a3ff9: No bootstrap required, opened a new log
I20240501 07:59:14.888492 12280 tablet_bootstrap.cc:492] T f2ce44c1d8c44b9789b21af5e889a419 P 83f20e84af4c40d09a4804b6b26e697f: No bootstrap required, opened a new log
I20240501 07:59:14.888581 12280 ts_tablet_manager.cc:1367] T f2ce44c1d8c44b9789b21af5e889a419 P 83f20e84af4c40d09a4804b6b26e697f: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 07:59:14.888581 12284 ts_tablet_manager.cc:1367] T f2ce44c1d8c44b9789b21af5e889a419 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 07:59:14.889060 12280 raft_consensus.cc:348] T f2ce44c1d8c44b9789b21af5e889a419 P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } }
I20240501 07:59:14.889205 12280 raft_consensus.cc:374] T f2ce44c1d8c44b9789b21af5e889a419 P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:14.889240 12280 raft_consensus.cc:729] T f2ce44c1d8c44b9789b21af5e889a419 P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83f20e84af4c40d09a4804b6b26e697f, State: Initialized, Role: FOLLOWER
I20240501 07:59:14.889369 12280 consensus_queue.cc:260] T f2ce44c1d8c44b9789b21af5e889a419 P 83f20e84af4c40d09a4804b6b26e697f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } }
I20240501 07:59:14.889647 12280 ts_tablet_manager.cc:1398] T f2ce44c1d8c44b9789b21af5e889a419 P 83f20e84af4c40d09a4804b6b26e697f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:14.890734 12284 raft_consensus.cc:348] T f2ce44c1d8c44b9789b21af5e889a419 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } }
I20240501 07:59:14.890866 12284 raft_consensus.cc:374] T f2ce44c1d8c44b9789b21af5e889a419 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:14.890902 12284 raft_consensus.cc:729] T f2ce44c1d8c44b9789b21af5e889a419 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b164cbbc00f4e8eb3254ddd349a3ff9, State: Initialized, Role: FOLLOWER
I20240501 07:59:14.891031 12284 consensus_queue.cc:260] T f2ce44c1d8c44b9789b21af5e889a419 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } }
I20240501 07:59:14.891321 12284 ts_tablet_manager.cc:1398] T f2ce44c1d8c44b9789b21af5e889a419 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 07:59:14.892194 12283 tablet_bootstrap.cc:492] T f2ce44c1d8c44b9789b21af5e889a419 P 74077b61d6f54342bf9736b4a9f2eade: No bootstrap required, opened a new log
I20240501 07:59:14.892292 12283 ts_tablet_manager.cc:1367] T f2ce44c1d8c44b9789b21af5e889a419 P 74077b61d6f54342bf9736b4a9f2eade: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240501 07:59:14.892833 12283 raft_consensus.cc:348] T f2ce44c1d8c44b9789b21af5e889a419 P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } }
I20240501 07:59:14.892947 12283 raft_consensus.cc:374] T f2ce44c1d8c44b9789b21af5e889a419 P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:14.893007 12283 raft_consensus.cc:729] T f2ce44c1d8c44b9789b21af5e889a419 P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74077b61d6f54342bf9736b4a9f2eade, State: Initialized, Role: FOLLOWER
I20240501 07:59:14.893141 12283 consensus_queue.cc:260] T f2ce44c1d8c44b9789b21af5e889a419 P 74077b61d6f54342bf9736b4a9f2eade [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } }
I20240501 07:59:14.893407 12283 ts_tablet_manager.cc:1398] T f2ce44c1d8c44b9789b21af5e889a419 P 74077b61d6f54342bf9736b4a9f2eade: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 07:59:14.906538 11719 raft_consensus.cc:1050] T 6daebcfc41754ec4aca939ca20588854 P 1f9cb72be98941b5a8e504cbe34132b9: attempting to promote NON_VOTER 8b8b3ad3aa8f4053b2167dd550f2086f to VOTER
I20240501 07:59:14.907001 11719 consensus_queue.cc:237] T 6daebcfc41754ec4aca939ca20588854 P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { promote: false } }
I20240501 07:59:14.908419 8925 raft_consensus.cc:1261] T 6daebcfc41754ec4aca939ca20588854 P 13d05ebdca8c4f489c597ce550dd773d [term 1 FOLLOWER]: Refusing update from remote peer 1f9cb72be98941b5a8e504cbe34132b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 07:59:14.908437 9056 raft_consensus.cc:1261] T 6daebcfc41754ec4aca939ca20588854 P a505e7cb133c4392bfda8a4a20874ae9 [term 1 FOLLOWER]: Refusing update from remote peer 1f9cb72be98941b5a8e504cbe34132b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 07:59:14.908515 6313 raft_consensus.cc:1261] T 6daebcfc41754ec4aca939ca20588854 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 LEARNER]: Refusing update from remote peer 1f9cb72be98941b5a8e504cbe34132b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 07:59:14.908851 12048 consensus_queue.cc:1035] T 6daebcfc41754ec4aca939ca20588854 P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 07:59:14.909117 12049 consensus_queue.cc:1035] T 6daebcfc41754ec4aca939ca20588854 P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 07:59:14.909358 11402 consensus_queue.cc:1035] T 6daebcfc41754ec4aca939ca20588854 P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 07:59:14.911155 8512 raft_consensus.cc:1050] T 6dc929e2a6ae45bd8b1f297583636bb6 P 8315d22c56f7439a9e1fc82f153f018c: attempting to promote NON_VOTER 83f20e84af4c40d09a4804b6b26e697f to VOTER
I20240501 07:59:14.911279 12048 raft_consensus.cc:2932] T 6daebcfc41754ec4aca939ca20588854 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { promote: false } } }
I20240501 07:59:14.911599 8512 consensus_queue.cc:237] T 6dc929e2a6ae45bd8b1f297583636bb6 P 8315d22c56f7439a9e1fc82f153f018c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false } }
I20240501 07:59:14.911700 8925 raft_consensus.cc:2932] T 6daebcfc41754ec4aca939ca20588854 P 13d05ebdca8c4f489c597ce550dd773d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { promote: false } } }
I20240501 07:59:14.913661 9189 raft_consensus.cc:1261] T 6dc929e2a6ae45bd8b1f297583636bb6 P 83f20e84af4c40d09a4804b6b26e697f [term 1 LEARNER]: Refusing update from remote peer 8315d22c56f7439a9e1fc82f153f018c: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 07:59:14.913661 8793 raft_consensus.cc:1261] T 6dc929e2a6ae45bd8b1f297583636bb6 P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Refusing update from remote peer 8315d22c56f7439a9e1fc82f153f018c: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 07:59:14.914151 11735 consensus_queue.cc:1035] T 6dc929e2a6ae45bd8b1f297583636bb6 P 8315d22c56f7439a9e1fc82f153f018c [LEADER]: Connected to new peer: Peer: permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 07:59:14.914140 9056 raft_consensus.cc:2932] T 6daebcfc41754ec4aca939ca20588854 P a505e7cb133c4392bfda8a4a20874ae9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { promote: false } } }
I20240501 07:59:14.914903 6313 raft_consensus.cc:1261] T 6dc929e2a6ae45bd8b1f297583636bb6 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Refusing update from remote peer 8315d22c56f7439a9e1fc82f153f018c: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 07:59:14.915391 11735 consensus_queue.cc:1035] T 6dc929e2a6ae45bd8b1f297583636bb6 P 8315d22c56f7439a9e1fc82f153f018c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 07:59:14.917232 6170 catalog_manager.cc:5484] T 6daebcfc41754ec4aca939ca20588854 P 13d05ebdca8c4f489c597ce550dd773d reported cstate change: config changed from index 2 to 3, 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1f9cb72be98941b5a8e504cbe34132b9" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { promote: false } } }
I20240501 07:59:14.917726 11735 consensus_queue.cc:1035] T 6dc929e2a6ae45bd8b1f297583636bb6 P 8315d22c56f7439a9e1fc82f153f018c [LEADER]: Connected to new peer: Peer: permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 07:59:14.918926 6311 raft_consensus.cc:2932] T 6daebcfc41754ec4aca939ca20588854 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { promote: false } } }
I20240501 07:59:14.920668 6707 consensus_queue.cc:237] T 6daebcfc41754ec4aca939ca20588854 P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { promote: false } }
I20240501 07:59:14.921850 12011 raft_consensus.cc:2932] T 6dc929e2a6ae45bd8b1f297583636bb6 P 8315d22c56f7439a9e1fc82f153f018c [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false } } }
I20240501 07:59:14.922314 6313 raft_consensus.cc:2932] T 6dc929e2a6ae45bd8b1f297583636bb6 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false } } }
I20240501 07:59:14.924683 9188 raft_consensus.cc:2932] T 6dc929e2a6ae45bd8b1f297583636bb6 P 83f20e84af4c40d09a4804b6b26e697f [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false } } }
I20240501 07:59:14.927619 6170 catalog_manager.cc:5484] T 6dc929e2a6ae45bd8b1f297583636bb6 P 8b8b3ad3aa8f4053b2167dd550f2086f reported cstate change: config changed from index 7 to 8, 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8315d22c56f7439a9e1fc82f153f018c" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false } } }
I20240501 07:59:14.928488 6313 raft_consensus.cc:1261] T 6daebcfc41754ec4aca939ca20588854 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Refusing update from remote peer 1f9cb72be98941b5a8e504cbe34132b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:59:14.928774 9056 raft_consensus.cc:1261] T 6daebcfc41754ec4aca939ca20588854 P a505e7cb133c4392bfda8a4a20874ae9 [term 1 FOLLOWER]: Refusing update from remote peer 1f9cb72be98941b5a8e504cbe34132b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:59:14.929239 11719 consensus_queue.cc:1035] T 6daebcfc41754ec4aca939ca20588854 P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:14.929602 11719 consensus_queue.cc:1035] T 6daebcfc41754ec4aca939ca20588854 P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:14.931438 8793 raft_consensus.cc:2932] T 6dc929e2a6ae45bd8b1f297583636bb6 P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false } } }
I20240501 07:59:14.933198 11719 raft_consensus.cc:2932] T 6daebcfc41754ec4aca939ca20588854 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { promote: false } } }
I20240501 07:59:14.934159 6313 raft_consensus.cc:2932] T 6daebcfc41754ec4aca939ca20588854 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { promote: false } } }
I20240501 07:59:14.934924 6159 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6daebcfc41754ec4aca939ca20588854 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 07:59:14.936386 9056 raft_consensus.cc:2932] T 6daebcfc41754ec4aca939ca20588854 P a505e7cb133c4392bfda8a4a20874ae9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { promote: false } } }
I20240501 07:59:14.937520 6170 catalog_manager.cc:5484] T 6daebcfc41754ec4aca939ca20588854 P 8b8b3ad3aa8f4053b2167dd550f2086f reported cstate change: config changed from index 3 to 4, VOTER 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134) evicted. New cstate: current_term: 1 leader_uuid: "1f9cb72be98941b5a8e504cbe34132b9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { promote: false } } }
I20240501 07:59:14.940232 12492 ts_tablet_manager.cc:902] T 28eaf22342ff4b5394f958d0b659273c P 3b164cbbc00f4e8eb3254ddd349a3ff9: Initiating tablet copy from peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607)
I20240501 07:59:14.940493 8904 tablet_service.cc:1514] Processing DeleteTablet for tablet 6daebcfc41754ec4aca939ca20588854 with delete_type TABLET_DATA_TOMBSTONED (TS 13d05ebdca8c4f489c597ce550dd773d not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:47202
I20240501 07:59:14.940743 12446 tablet_replica.cc:330] T 6daebcfc41754ec4aca939ca20588854 P 13d05ebdca8c4f489c597ce550dd773d: stopping tablet replica
I20240501 07:59:14.940902 12446 raft_consensus.cc:2229] T 6daebcfc41754ec4aca939ca20588854 P 13d05ebdca8c4f489c597ce550dd773d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:14.941077 12492 tablet_copy_client.cc:308] T 28eaf22342ff4b5394f958d0b659273c P 3b164cbbc00f4e8eb3254ddd349a3ff9: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.135:42607
I20240501 07:59:14.941094 12446 raft_consensus.cc:2258] T 6daebcfc41754ec4aca939ca20588854 P 13d05ebdca8c4f489c597ce550dd773d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:14.941552 9076 tablet_copy_service.cc:140] P a505e7cb133c4392bfda8a4a20874ae9: Received BeginTabletCopySession request for tablet 28eaf22342ff4b5394f958d0b659273c from peer 3b164cbbc00f4e8eb3254ddd349a3ff9 ({username='slave'} at 127.0.99.131:39545)
I20240501 07:59:14.941638 9076 tablet_copy_service.cc:161] P a505e7cb133c4392bfda8a4a20874ae9: Beginning new tablet copy session on tablet 28eaf22342ff4b5394f958d0b659273c from peer 3b164cbbc00f4e8eb3254ddd349a3ff9 at {username='slave'} at 127.0.99.131:39545: session id = 3b164cbbc00f4e8eb3254ddd349a3ff9-28eaf22342ff4b5394f958d0b659273c
I20240501 07:59:14.941699 12446 ts_tablet_manager.cc:1875] T 6daebcfc41754ec4aca939ca20588854 P 13d05ebdca8c4f489c597ce550dd773d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:59:14.942353 9076 tablet_copy_source_session.cc:205] T 28eaf22342ff4b5394f958d0b659273c P a505e7cb133c4392bfda8a4a20874ae9: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:14.943121 12492 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28eaf22342ff4b5394f958d0b659273c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:14.943424 12446 ts_tablet_manager.cc:1888] T 6daebcfc41754ec4aca939ca20588854 P 13d05ebdca8c4f489c597ce550dd773d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 07:59:14.943491 12446 log.cc:1196] T 6daebcfc41754ec4aca939ca20588854 P 13d05ebdca8c4f489c597ce550dd773d: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/6daebcfc41754ec4aca939ca20588854
I20240501 07:59:14.943954 6159 catalog_manager.cc:4830] TS 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134:40141): tablet 6daebcfc41754ec4aca939ca20588854 (table rebalancer_extra_table_12 [id=731fe27ae8e34d92bd595d43c4bc8f3a]) successfully deleted
I20240501 07:59:14.945079 12492 tablet_copy_client.cc:783] T 28eaf22342ff4b5394f958d0b659273c P 3b164cbbc00f4e8eb3254ddd349a3ff9: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:14.945230 12492 tablet_copy_client.cc:647] T 28eaf22342ff4b5394f958d0b659273c P 3b164cbbc00f4e8eb3254ddd349a3ff9: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:14.946309 12492 tablet_copy_client.cc:523] T 28eaf22342ff4b5394f958d0b659273c P 3b164cbbc00f4e8eb3254ddd349a3ff9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:14.947307 12492 tablet_bootstrap.cc:492] T 28eaf22342ff4b5394f958d0b659273c P 3b164cbbc00f4e8eb3254ddd349a3ff9: Bootstrap starting.
I20240501 07:59:14.951094 12492 tablet_bootstrap.cc:492] T 28eaf22342ff4b5394f958d0b659273c P 3b164cbbc00f4e8eb3254ddd349a3ff9: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:14.951324 12492 tablet_bootstrap.cc:492] T 28eaf22342ff4b5394f958d0b659273c P 3b164cbbc00f4e8eb3254ddd349a3ff9: Bootstrap complete.
I20240501 07:59:14.951476 12492 ts_tablet_manager.cc:1367] T 28eaf22342ff4b5394f958d0b659273c P 3b164cbbc00f4e8eb3254ddd349a3ff9: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20240501 07:59:14.951900 12492 raft_consensus.cc:348] T 28eaf22342ff4b5394f958d0b659273c P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: true } }
I20240501 07:59:14.952032 12492 raft_consensus.cc:729] T 28eaf22342ff4b5394f958d0b659273c P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3b164cbbc00f4e8eb3254ddd349a3ff9, State: Initialized, Role: LEARNER
I20240501 07:59:14.952155 12492 consensus_queue.cc:260] T 28eaf22342ff4b5394f958d0b659273c P 3b164cbbc00f4e8eb3254ddd349a3ff9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: true } }
I20240501 07:59:14.952425 12492 ts_tablet_manager.cc:1398] T 28eaf22342ff4b5394f958d0b659273c P 3b164cbbc00f4e8eb3254ddd349a3ff9: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:59:14.953506 9076 tablet_copy_service.cc:342] P a505e7cb133c4392bfda8a4a20874ae9: Request end of tablet copy session 3b164cbbc00f4e8eb3254ddd349a3ff9-28eaf22342ff4b5394f958d0b659273c received from {username='slave'} at 127.0.99.131:39545
I20240501 07:59:14.953596 9076 tablet_copy_service.cc:433] P a505e7cb133c4392bfda8a4a20874ae9: ending tablet copy session 3b164cbbc00f4e8eb3254ddd349a3ff9-28eaf22342ff4b5394f958d0b659273c on tablet 28eaf22342ff4b5394f958d0b659273c with peer 3b164cbbc00f4e8eb3254ddd349a3ff9
I20240501 07:59:14.968875 12183 raft_consensus.cc:482] T f2ce44c1d8c44b9789b21af5e889a419 P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:14.969072 12183 raft_consensus.cc:504] T f2ce44c1d8c44b9789b21af5e889a419 P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } }
I20240501 07:59:14.969503 12183 leader_election.cc:288] T f2ce44c1d8c44b9789b21af5e889a419 P 83f20e84af4c40d09a4804b6b26e697f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131:34471), 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133:39859)
I20240501 07:59:14.970002 8793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2ce44c1d8c44b9789b21af5e889a419" candidate_uuid: "83f20e84af4c40d09a4804b6b26e697f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74077b61d6f54342bf9736b4a9f2eade" is_pre_election: true
I20240501 07:59:14.970069 12468 ts_tablet_manager.cc:902] T a4727883b2544ed58b356f41dc8017a9 P 13d05ebdca8c4f489c597ce550dd773d: Initiating tablet copy from peer 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131:34471)
I20240501 07:59:14.970146 8793 raft_consensus.cc:2454] T f2ce44c1d8c44b9789b21af5e889a419 P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83f20e84af4c40d09a4804b6b26e697f in term 0.
I20240501 07:59:14.970320 8512 raft_consensus.cc:1050] T 54b3efabad7f4b608cc9d931546b5743 P 8315d22c56f7439a9e1fc82f153f018c: attempting to promote NON_VOTER 8b8b3ad3aa8f4053b2167dd550f2086f to VOTER
I20240501 07:59:14.970458 9122 leader_election.cc:302] T f2ce44c1d8c44b9789b21af5e889a419 P 83f20e84af4c40d09a4804b6b26e697f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74077b61d6f54342bf9736b4a9f2eade, 83f20e84af4c40d09a4804b6b26e697f; no voters:
I20240501 07:59:14.970546 12468 tablet_copy_client.cc:308] T a4727883b2544ed58b356f41dc8017a9 P 13d05ebdca8c4f489c597ce550dd773d: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:34471
I20240501 07:59:14.970641 12183 raft_consensus.cc:2781] T f2ce44c1d8c44b9789b21af5e889a419 P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:14.970700 12183 raft_consensus.cc:482] T f2ce44c1d8c44b9789b21af5e889a419 P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:14.970737 12183 raft_consensus.cc:3037] T f2ce44c1d8c44b9789b21af5e889a419 P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:14.970752 8512 consensus_queue.cc:237] T 54b3efabad7f4b608cc9d931546b5743 P 8315d22c56f7439a9e1fc82f153f018c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { replace: true } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { promote: false } }
I20240501 07:59:14.971369 6597 tablet_copy_service.cc:140] P 3b164cbbc00f4e8eb3254ddd349a3ff9: Received BeginTabletCopySession request for tablet a4727883b2544ed58b356f41dc8017a9 from peer 13d05ebdca8c4f489c597ce550dd773d ({username='slave'} at 127.0.99.134:42357)
I20240501 07:59:14.971473 6597 tablet_copy_service.cc:161] P 3b164cbbc00f4e8eb3254ddd349a3ff9: Beginning new tablet copy session on tablet a4727883b2544ed58b356f41dc8017a9 from peer 13d05ebdca8c4f489c597ce550dd773d at {username='slave'} at 127.0.99.134:42357: session id = 13d05ebdca8c4f489c597ce550dd773d-a4727883b2544ed58b356f41dc8017a9
I20240501 07:59:14.971863 12183 raft_consensus.cc:504] T f2ce44c1d8c44b9789b21af5e889a419 P 83f20e84af4c40d09a4804b6b26e697f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } }
I20240501 07:59:14.972201 12183 leader_election.cc:288] T f2ce44c1d8c44b9789b21af5e889a419 P 83f20e84af4c40d09a4804b6b26e697f [CANDIDATE]: Term 1 election: Requested vote from peers 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131:34471), 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133:39859)
I20240501 07:59:14.972251 9189 raft_consensus.cc:1261] T 54b3efabad7f4b608cc9d931546b5743 P 83f20e84af4c40d09a4804b6b26e697f [term 1 FOLLOWER]: Refusing update from remote peer 8315d22c56f7439a9e1fc82f153f018c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 07:59:14.972261 6707 raft_consensus.cc:1261] T 54b3efabad7f4b608cc9d931546b5743 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Refusing update from remote peer 8315d22c56f7439a9e1fc82f153f018c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 07:59:14.972558 6597 tablet_copy_source_session.cc:205] T a4727883b2544ed58b356f41dc8017a9 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:14.972585 8793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2ce44c1d8c44b9789b21af5e889a419" candidate_uuid: "83f20e84af4c40d09a4804b6b26e697f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74077b61d6f54342bf9736b4a9f2eade"
I20240501 07:59:14.972694 8793 raft_consensus.cc:3037] T f2ce44c1d8c44b9789b21af5e889a419 P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:14.972795 12011 consensus_queue.cc:1035] T 54b3efabad7f4b608cc9d931546b5743 P 8315d22c56f7439a9e1fc82f153f018c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 07:59:14.973186 12011 consensus_queue.cc:1035] T 54b3efabad7f4b608cc9d931546b5743 P 8315d22c56f7439a9e1fc82f153f018c [LEADER]: Connected to new peer: Peer: permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 07:59:14.973230 6577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2ce44c1d8c44b9789b21af5e889a419" candidate_uuid: "83f20e84af4c40d09a4804b6b26e697f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9"
I20240501 07:59:14.973332 6577 raft_consensus.cc:3037] T f2ce44c1d8c44b9789b21af5e889a419 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:14.973764 12468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4727883b2544ed58b356f41dc8017a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:14.973768 8793 raft_consensus.cc:2454] T f2ce44c1d8c44b9789b21af5e889a419 P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83f20e84af4c40d09a4804b6b26e697f in term 1.
I20240501 07:59:14.974004 6575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2ce44c1d8c44b9789b21af5e889a419" candidate_uuid: "83f20e84af4c40d09a4804b6b26e697f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" is_pre_election: true
I20240501 07:59:14.974104 9122 leader_election.cc:302] T f2ce44c1d8c44b9789b21af5e889a419 P 83f20e84af4c40d09a4804b6b26e697f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74077b61d6f54342bf9736b4a9f2eade, 83f20e84af4c40d09a4804b6b26e697f; no voters:
I20240501 07:59:14.974282 12183 raft_consensus.cc:2781] T f2ce44c1d8c44b9789b21af5e889a419 P 83f20e84af4c40d09a4804b6b26e697f [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:14.974400 6577 raft_consensus.cc:2454] T f2ce44c1d8c44b9789b21af5e889a419 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83f20e84af4c40d09a4804b6b26e697f in term 1.
I20240501 07:59:14.974476 12183 raft_consensus.cc:686] T f2ce44c1d8c44b9789b21af5e889a419 P 83f20e84af4c40d09a4804b6b26e697f [term 1 LEADER]: Becoming Leader. State: Replica: 83f20e84af4c40d09a4804b6b26e697f, State: Running, Role: LEADER
I20240501 07:59:14.974622 12183 consensus_queue.cc:237] T f2ce44c1d8c44b9789b21af5e889a419 P 83f20e84af4c40d09a4804b6b26e697f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } }
I20240501 07:59:14.975986 12468 tablet_copy_client.cc:783] T a4727883b2544ed58b356f41dc8017a9 P 13d05ebdca8c4f489c597ce550dd773d: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:14.976143 12468 tablet_copy_client.cc:647] T a4727883b2544ed58b356f41dc8017a9 P 13d05ebdca8c4f489c597ce550dd773d: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:14.976645 6313 raft_consensus.cc:1261] T 54b3efabad7f4b608cc9d931546b5743 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 LEARNER]: Refusing update from remote peer 8315d22c56f7439a9e1fc82f153f018c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 07:59:14.977002 6171 catalog_manager.cc:5484] T f2ce44c1d8c44b9789b21af5e889a419 P 83f20e84af4c40d09a4804b6b26e697f reported cstate change: term changed from 0 to 1, leader changed from <none> to 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136). New cstate: current_term: 1 leader_uuid: "83f20e84af4c40d09a4804b6b26e697f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } health_report { overall_health: UNKNOWN } } }
I20240501 07:59:14.977156 10111 consensus_queue.cc:1035] T 54b3efabad7f4b608cc9d931546b5743 P 8315d22c56f7439a9e1fc82f153f018c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 07:59:14.978127 10111 raft_consensus.cc:2932] T 54b3efabad7f4b608cc9d931546b5743 P 8315d22c56f7439a9e1fc82f153f018c [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { replace: true } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { promote: false } } }
I20240501 07:59:14.978245 12468 tablet_copy_client.cc:523] T a4727883b2544ed58b356f41dc8017a9 P 13d05ebdca8c4f489c597ce550dd773d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:14.980157 12468 tablet_bootstrap.cc:492] T a4727883b2544ed58b356f41dc8017a9 P 13d05ebdca8c4f489c597ce550dd773d: Bootstrap starting.
I20240501 07:59:14.980779 9189 raft_consensus.cc:2932] T 54b3efabad7f4b608cc9d931546b5743 P 83f20e84af4c40d09a4804b6b26e697f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { replace: true } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { promote: false } } }
I20240501 07:59:14.980754 6707 raft_consensus.cc:2932] T 54b3efabad7f4b608cc9d931546b5743 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { replace: true } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { promote: false } } }
I20240501 07:59:14.980748 6313 raft_consensus.cc:2932] T 54b3efabad7f4b608cc9d931546b5743 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { replace: true } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { promote: false } } }
I20240501 07:59:14.981190 6171 catalog_manager.cc:5484] T 54b3efabad7f4b608cc9d931546b5743 P 8315d22c56f7439a9e1fc82f153f018c reported cstate change: config changed from index 2 to 3, 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8315d22c56f7439a9e1fc82f153f018c" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 07:59:14.987289 6444 consensus_queue.cc:237] T 54b3efabad7f4b608cc9d931546b5743 P 8315d22c56f7439a9e1fc82f153f018c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { promote: false } }
I20240501 07:59:14.988411 6707 raft_consensus.cc:1261] T 54b3efabad7f4b608cc9d931546b5743 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Refusing update from remote peer 8315d22c56f7439a9e1fc82f153f018c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:59:14.988577 6313 raft_consensus.cc:1261] T 54b3efabad7f4b608cc9d931546b5743 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Refusing update from remote peer 8315d22c56f7439a9e1fc82f153f018c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:59:14.988910 12011 consensus_queue.cc:1035] T 54b3efabad7f4b608cc9d931546b5743 P 8315d22c56f7439a9e1fc82f153f018c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:14.989122 8512 consensus_queue.cc:1035] T 54b3efabad7f4b608cc9d931546b5743 P 8315d22c56f7439a9e1fc82f153f018c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:14.990656 12011 raft_consensus.cc:2932] T 54b3efabad7f4b608cc9d931546b5743 P 8315d22c56f7439a9e1fc82f153f018c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { promote: false } } }
I20240501 07:59:14.991894 6707 raft_consensus.cc:2932] T 54b3efabad7f4b608cc9d931546b5743 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { promote: false } } }
I20240501 07:59:14.993332 6159 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 54b3efabad7f4b608cc9d931546b5743 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 07:59:14.994150 6313 raft_consensus.cc:2932] T 54b3efabad7f4b608cc9d931546b5743 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { promote: false } } }
I20240501 07:59:14.994637 6170 catalog_manager.cc:5484] T 54b3efabad7f4b608cc9d931546b5743 P 8315d22c56f7439a9e1fc82f153f018c reported cstate change: config changed from index 3 to 4, VOTER 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136) evicted. New cstate: current_term: 1 leader_uuid: "8315d22c56f7439a9e1fc82f153f018c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 07:59:14.996544 12123 consensus_queue.cc:1035] T 0f9e2549fe294a93b3b25081a15b2028 P 74077b61d6f54342bf9736b4a9f2eade [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:14.998587 9169 tablet_service.cc:1514] Processing DeleteTablet for tablet 54b3efabad7f4b608cc9d931546b5743 with delete_type TABLET_DATA_TOMBSTONED (TS 83f20e84af4c40d09a4804b6b26e697f not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:33594
I20240501 07:59:14.999013 12510 tablet_replica.cc:330] T 54b3efabad7f4b608cc9d931546b5743 P 83f20e84af4c40d09a4804b6b26e697f: stopping tablet replica
I20240501 07:59:14.999194 12510 raft_consensus.cc:2229] T 54b3efabad7f4b608cc9d931546b5743 P 83f20e84af4c40d09a4804b6b26e697f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:14.999370 12510 raft_consensus.cc:2258] T 54b3efabad7f4b608cc9d931546b5743 P 83f20e84af4c40d09a4804b6b26e697f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:15.000149 12510 ts_tablet_manager.cc:1875] T 54b3efabad7f4b608cc9d931546b5743 P 83f20e84af4c40d09a4804b6b26e697f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:59:15.005717 12123 consensus_queue.cc:1035] T 0f9e2549fe294a93b3b25081a15b2028 P 74077b61d6f54342bf9736b4a9f2eade [LEADER]: Connected to new peer: Peer: permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:15.011682 12468 tablet_bootstrap.cc:492] T a4727883b2544ed58b356f41dc8017a9 P 13d05ebdca8c4f489c597ce550dd773d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:15.012073 12468 tablet_bootstrap.cc:492] T a4727883b2544ed58b356f41dc8017a9 P 13d05ebdca8c4f489c597ce550dd773d: Bootstrap complete.
I20240501 07:59:15.012284 12468 ts_tablet_manager.cc:1367] T a4727883b2544ed58b356f41dc8017a9 P 13d05ebdca8c4f489c597ce550dd773d: Time spent bootstrapping tablet: real 0.032s user 0.022s sys 0.004s
I20240501 07:59:15.012792 12468 raft_consensus.cc:348] T a4727883b2544ed58b356f41dc8017a9 P 13d05ebdca8c4f489c597ce550dd773d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: true } }
I20240501 07:59:15.012920 12468 raft_consensus.cc:729] T a4727883b2544ed58b356f41dc8017a9 P 13d05ebdca8c4f489c597ce550dd773d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 13d05ebdca8c4f489c597ce550dd773d, State: Initialized, Role: LEARNER
I20240501 07:59:15.013038 12468 consensus_queue.cc:260] T a4727883b2544ed58b356f41dc8017a9 P 13d05ebdca8c4f489c597ce550dd773d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: true } }
I20240501 07:59:15.013922 12468 ts_tablet_manager.cc:1398] T a4727883b2544ed58b356f41dc8017a9 P 13d05ebdca8c4f489c597ce550dd773d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:59:15.014432 6597 tablet_copy_service.cc:342] P 3b164cbbc00f4e8eb3254ddd349a3ff9: Request end of tablet copy session 13d05ebdca8c4f489c597ce550dd773d-a4727883b2544ed58b356f41dc8017a9 received from {username='slave'} at 127.0.99.134:42357
I20240501 07:59:15.014513 6597 tablet_copy_service.cc:433] P 3b164cbbc00f4e8eb3254ddd349a3ff9: ending tablet copy session 13d05ebdca8c4f489c597ce550dd773d-a4727883b2544ed58b356f41dc8017a9 on tablet a4727883b2544ed58b356f41dc8017a9 with peer 13d05ebdca8c4f489c597ce550dd773d
I20240501 07:59:15.017449 12510 ts_tablet_manager.cc:1888] T 54b3efabad7f4b608cc9d931546b5743 P 83f20e84af4c40d09a4804b6b26e697f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 07:59:15.017536 12510 log.cc:1196] T 54b3efabad7f4b608cc9d931546b5743 P 83f20e84af4c40d09a4804b6b26e697f: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/54b3efabad7f4b608cc9d931546b5743
I20240501 07:59:15.018226 6158 catalog_manager.cc:4830] TS 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): tablet 54b3efabad7f4b608cc9d931546b5743 (table rebalancer_extra_table_2 [id=5e1db4e06c5e470a9d2f8a20dd23c585]) successfully deleted
I20240501 07:59:15.035034 11901 raft_consensus.cc:1050] T 2e540f389b2a4aef93a020f549d11b67 P 8b8b3ad3aa8f4053b2167dd550f2086f: attempting to promote NON_VOTER 13d05ebdca8c4f489c597ce550dd773d to VOTER
I20240501 07:59:15.035475 11901 consensus_queue.cc:237] T 2e540f389b2a4aef93a020f549d11b67 P 8b8b3ad3aa8f4053b2167dd550f2086f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false } }
I20240501 07:59:15.036863 9189 raft_consensus.cc:1261] T 2e540f389b2a4aef93a020f549d11b67 P 83f20e84af4c40d09a4804b6b26e697f [term 1 FOLLOWER]: Refusing update from remote peer 8b8b3ad3aa8f4053b2167dd550f2086f: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 07:59:15.036875 6577 raft_consensus.cc:1261] T 2e540f389b2a4aef93a020f549d11b67 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Refusing update from remote peer 8b8b3ad3aa8f4053b2167dd550f2086f: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 07:59:15.037344 12159 consensus_queue.cc:1035] T 2e540f389b2a4aef93a020f549d11b67 P 8b8b3ad3aa8f4053b2167dd550f2086f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 07:59:15.037490 8925 raft_consensus.cc:1261] T 2e540f389b2a4aef93a020f549d11b67 P 13d05ebdca8c4f489c597ce550dd773d [term 1 LEARNER]: Refusing update from remote peer 8b8b3ad3aa8f4053b2167dd550f2086f: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 07:59:15.037673 12159 consensus_queue.cc:1035] T 2e540f389b2a4aef93a020f549d11b67 P 8b8b3ad3aa8f4053b2167dd550f2086f [LEADER]: Connected to new peer: Peer: permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 07:59:15.038719 11094 consensus_queue.cc:1035] T 2e540f389b2a4aef93a020f549d11b67 P 8b8b3ad3aa8f4053b2167dd550f2086f [LEADER]: Connected to new peer: Peer: permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 07:59:15.040799 11901 raft_consensus.cc:2932] T 2e540f389b2a4aef93a020f549d11b67 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false } } }
I20240501 07:59:15.041224 6577 raft_consensus.cc:2932] T 2e540f389b2a4aef93a020f549d11b67 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false } } }
I20240501 07:59:15.043346 9189 raft_consensus.cc:2932] T 2e540f389b2a4aef93a020f549d11b67 P 83f20e84af4c40d09a4804b6b26e697f [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false } } }
I20240501 07:59:15.043972 6172 catalog_manager.cc:5484] T 2e540f389b2a4aef93a020f549d11b67 P 8b8b3ad3aa8f4053b2167dd550f2086f reported cstate change: config changed from index 7 to 8, 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 07:59:15.051988 8925 raft_consensus.cc:2932] T 2e540f389b2a4aef93a020f549d11b67 P 13d05ebdca8c4f489c597ce550dd773d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false } } }
I20240501 07:59:15.052644 6313 consensus_queue.cc:237] T 2e540f389b2a4aef93a020f549d11b67 P 8b8b3ad3aa8f4053b2167dd550f2086f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false } }
I20240501 07:59:15.054245 9189 raft_consensus.cc:1261] T 2e540f389b2a4aef93a020f549d11b67 P 83f20e84af4c40d09a4804b6b26e697f [term 1 FOLLOWER]: Refusing update from remote peer 8b8b3ad3aa8f4053b2167dd550f2086f: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 07:59:15.054750 12159 consensus_queue.cc:1035] T 2e540f389b2a4aef93a020f549d11b67 P 8b8b3ad3aa8f4053b2167dd550f2086f [LEADER]: Connected to new peer: Peer: permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 07:59:15.054867 8925 raft_consensus.cc:1261] T 2e540f389b2a4aef93a020f549d11b67 P 13d05ebdca8c4f489c597ce550dd773d [term 1 FOLLOWER]: Refusing update from remote peer 8b8b3ad3aa8f4053b2167dd550f2086f: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 07:59:15.055411 12159 consensus_queue.cc:1035] T 2e540f389b2a4aef93a020f549d11b67 P 8b8b3ad3aa8f4053b2167dd550f2086f [LEADER]: Connected to new peer: Peer: permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 07:59:15.057220 11094 raft_consensus.cc:2932] T 2e540f389b2a4aef93a020f549d11b67 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false } } }
I20240501 07:59:15.057808 9189 raft_consensus.cc:2932] T 2e540f389b2a4aef93a020f549d11b67 P 83f20e84af4c40d09a4804b6b26e697f [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false } } }
I20240501 07:59:15.058260 8925 raft_consensus.cc:2932] T 2e540f389b2a4aef93a020f549d11b67 P 13d05ebdca8c4f489c597ce550dd773d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false } } }
I20240501 07:59:15.060364 6157 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 2e540f389b2a4aef93a020f549d11b67 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 07:59:15.061133 6171 catalog_manager.cc:5484] T 2e540f389b2a4aef93a020f549d11b67 P 8b8b3ad3aa8f4053b2167dd550f2086f reported cstate change: config changed from index 8 to 9, VOTER 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 07:59:15.061688 12391 ts_tablet_manager.cc:902] T 6221deb431d94bfca1c14696e0e0a0c5 P 74077b61d6f54342bf9736b4a9f2eade: Initiating tablet copy from peer 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132:36773)
I20240501 07:59:15.062147 12391 tablet_copy_client.cc:308] T 6221deb431d94bfca1c14696e0e0a0c5 P 74077b61d6f54342bf9736b4a9f2eade: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:36773
I20240501 07:59:15.062626 6728 tablet_copy_service.cc:140] P 1f9cb72be98941b5a8e504cbe34132b9: Received BeginTabletCopySession request for tablet 6221deb431d94bfca1c14696e0e0a0c5 from peer 74077b61d6f54342bf9736b4a9f2eade ({username='slave'} at 127.0.99.133:46973)
I20240501 07:59:15.062736 6728 tablet_copy_service.cc:161] P 1f9cb72be98941b5a8e504cbe34132b9: Beginning new tablet copy session on tablet 6221deb431d94bfca1c14696e0e0a0c5 from peer 74077b61d6f54342bf9736b4a9f2eade at {username='slave'} at 127.0.99.133:46973: session id = 74077b61d6f54342bf9736b4a9f2eade-6221deb431d94bfca1c14696e0e0a0c5
I20240501 07:59:15.063668 6728 tablet_copy_source_session.cc:205] T 6221deb431d94bfca1c14696e0e0a0c5 P 1f9cb72be98941b5a8e504cbe34132b9: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:15.065263 6556 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e540f389b2a4aef93a020f549d11b67 with delete_type TABLET_DATA_TOMBSTONED (TS 3b164cbbc00f4e8eb3254ddd349a3ff9 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:53874
I20240501 07:59:15.065371 12391 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6221deb431d94bfca1c14696e0e0a0c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:15.065479 12354 tablet_replica.cc:330] T 2e540f389b2a4aef93a020f549d11b67 P 3b164cbbc00f4e8eb3254ddd349a3ff9: stopping tablet replica
I20240501 07:59:15.065608 12354 raft_consensus.cc:2229] T 2e540f389b2a4aef93a020f549d11b67 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:15.065819 12354 raft_consensus.cc:2258] T 2e540f389b2a4aef93a020f549d11b67 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:15.066540 12354 ts_tablet_manager.cc:1875] T 2e540f389b2a4aef93a020f549d11b67 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:59:15.067353 12391 tablet_copy_client.cc:783] T 6221deb431d94bfca1c14696e0e0a0c5 P 74077b61d6f54342bf9736b4a9f2eade: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:15.067504 12391 tablet_copy_client.cc:647] T 6221deb431d94bfca1c14696e0e0a0c5 P 74077b61d6f54342bf9736b4a9f2eade: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:15.068607 12354 ts_tablet_manager.cc:1888] T 2e540f389b2a4aef93a020f549d11b67 P 3b164cbbc00f4e8eb3254ddd349a3ff9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240501 07:59:15.068672 12354 log.cc:1196] T 2e540f389b2a4aef93a020f549d11b67 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/2e540f389b2a4aef93a020f549d11b67
I20240501 07:59:15.069883 12391 tablet_copy_client.cc:523] T 6221deb431d94bfca1c14696e0e0a0c5 P 74077b61d6f54342bf9736b4a9f2eade: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:15.070819 6160 catalog_manager.cc:4830] TS 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131:34471): tablet 2e540f389b2a4aef93a020f549d11b67 (table rebalance_test_table_15 [id=ecf5cb7235314f20bbb522737ba7fd3c]) successfully deleted
I20240501 07:59:15.071187 12391 tablet_bootstrap.cc:492] T 6221deb431d94bfca1c14696e0e0a0c5 P 74077b61d6f54342bf9736b4a9f2eade: Bootstrap starting.
I20240501 07:59:15.090341 12123 consensus_queue.cc:1035] T 36aaf192042f4320b2526322424dc640 P 74077b61d6f54342bf9736b4a9f2eade [LEADER]: Connected to new peer: Peer: permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:15.095249 12123 consensus_queue.cc:1035] T 36aaf192042f4320b2526322424dc640 P 74077b61d6f54342bf9736b4a9f2eade [LEADER]: Connected to new peer: Peer: permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:15.099499 12391 tablet_bootstrap.cc:492] T 6221deb431d94bfca1c14696e0e0a0c5 P 74077b61d6f54342bf9736b4a9f2eade: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:15.099828 12391 tablet_bootstrap.cc:492] T 6221deb431d94bfca1c14696e0e0a0c5 P 74077b61d6f54342bf9736b4a9f2eade: Bootstrap complete.
I20240501 07:59:15.100042 12391 ts_tablet_manager.cc:1367] T 6221deb431d94bfca1c14696e0e0a0c5 P 74077b61d6f54342bf9736b4a9f2eade: Time spent bootstrapping tablet: real 0.029s user 0.025s sys 0.000s
I20240501 07:59:15.100468 12391 raft_consensus.cc:348] T 6221deb431d94bfca1c14696e0e0a0c5 P 74077b61d6f54342bf9736b4a9f2eade [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: true } }
I20240501 07:59:15.100606 12391 raft_consensus.cc:729] T 6221deb431d94bfca1c14696e0e0a0c5 P 74077b61d6f54342bf9736b4a9f2eade [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 74077b61d6f54342bf9736b4a9f2eade, State: Initialized, Role: LEARNER
I20240501 07:59:15.100935 12391 consensus_queue.cc:260] T 6221deb431d94bfca1c14696e0e0a0c5 P 74077b61d6f54342bf9736b4a9f2eade [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: true } }
I20240501 07:59:15.101150 6166 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55950:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 07:59:15.101397 12391 ts_tablet_manager.cc:1398] T 6221deb431d94bfca1c14696e0e0a0c5 P 74077b61d6f54342bf9736b4a9f2eade: Time spent starting tablet: real 0.001s user 0.000s sys 0.003s
I20240501 07:59:15.102016 6728 tablet_copy_service.cc:342] P 1f9cb72be98941b5a8e504cbe34132b9: Request end of tablet copy session 74077b61d6f54342bf9736b4a9f2eade-6221deb431d94bfca1c14696e0e0a0c5 received from {username='slave'} at 127.0.99.133:46973
I20240501 07:59:15.102113 6728 tablet_copy_service.cc:433] P 1f9cb72be98941b5a8e504cbe34132b9: ending tablet copy session 74077b61d6f54342bf9736b4a9f2eade-6221deb431d94bfca1c14696e0e0a0c5 on tablet 6221deb431d94bfca1c14696e0e0a0c5 with peer 74077b61d6f54342bf9736b4a9f2eade
I20240501 07:59:15.106143 8793 raft_consensus.cc:1261] T f2ce44c1d8c44b9789b21af5e889a419 P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Refusing update from remote peer 83f20e84af4c40d09a4804b6b26e697f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:15.106212 6577 raft_consensus.cc:1261] T f2ce44c1d8c44b9789b21af5e889a419 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Refusing update from remote peer 83f20e84af4c40d09a4804b6b26e697f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:15.106663 12183 consensus_queue.cc:1035] T f2ce44c1d8c44b9789b21af5e889a419 P 83f20e84af4c40d09a4804b6b26e697f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:15.106850 12503 consensus_queue.cc:1035] T f2ce44c1d8c44b9789b21af5e889a419 P 83f20e84af4c40d09a4804b6b26e697f [LEADER]: Connected to new peer: Peer: permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:15.110124 6166 catalog_manager.cc:5576] TS 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131:34471) does not have the latest schema for tablet f2ce44c1d8c44b9789b21af5e889a419 (table rebalancer_dynamic_table [id=956c08c49bed4f7ea842f14bdfec9e03]). Expected version 1 got 0
I20240501 07:59:15.111639 6170 catalog_manager.cc:5576] TS 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149) does not have the latest schema for tablet f2ce44c1d8c44b9789b21af5e889a419 (table rebalancer_dynamic_table [id=956c08c49bed4f7ea842f14bdfec9e03]). Expected version 1 got 0
I20240501 07:59:15.112376 6166 catalog_manager.cc:5576] TS 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133:39859) does not have the latest schema for tablet f2ce44c1d8c44b9789b21af5e889a419 (table rebalancer_dynamic_table [id=956c08c49bed4f7ea842f14bdfec9e03]). Expected version 1 got 0
I20240501 07:59:15.113108 12574 tablet.cc:1722] T f2ce44c1d8c44b9789b21af5e889a419 P 83f20e84af4c40d09a4804b6b26e697f: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 07:59:15.113557 12574 tablet.cc:1620] T f2ce44c1d8c44b9789b21af5e889a419 P 83f20e84af4c40d09a4804b6b26e697f: MemRowSet was empty: no flush needed.
I20240501 07:59:15.114987 6166 catalog_manager.cc:5576] TS 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133:39859) does not have the latest schema for tablet f2ce44c1d8c44b9789b21af5e889a419 (table rebalancer_dynamic_table [id=956c08c49bed4f7ea842f14bdfec9e03]). Expected version 1 got 0
I20240501 07:59:15.115483 12575 tablet.cc:1722] T f2ce44c1d8c44b9789b21af5e889a419 P 74077b61d6f54342bf9736b4a9f2eade: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 07:59:15.115967 12575 tablet.cc:1620] T f2ce44c1d8c44b9789b21af5e889a419 P 74077b61d6f54342bf9736b4a9f2eade: MemRowSet was empty: no flush needed.
I20240501 07:59:15.116679 12576 tablet.cc:1722] T f2ce44c1d8c44b9789b21af5e889a419 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 07:59:15.117992 6171 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 0bc706e85ce74cef97a4827ea4d4aa65: rebalancer_dynamic_table [id=956c08c49bed4f7ea842f14bdfec9e03] alter complete (version 1)
I20240501 07:59:15.118726 6166 catalog_manager.cc:5576] TS 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131:34471) does not have the latest schema for tablet f2ce44c1d8c44b9789b21af5e889a419 (table rebalancer_dynamic_table [id=956c08c49bed4f7ea842f14bdfec9e03]). Expected version 1 got 0
I20240501 07:59:15.119143 12576 tablet.cc:1620] T f2ce44c1d8c44b9789b21af5e889a419 P 3b164cbbc00f4e8eb3254ddd349a3ff9: MemRowSet was empty: no flush needed.
I20240501 07:59:15.135993 6444 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6dc929e2a6ae45bd8b1f297583636bb6"
dest_uuid: "8315d22c56f7439a9e1fc82f153f018c"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:59450
I20240501 07:59:15.136137 6444 raft_consensus.cc:595] T 6dc929e2a6ae45bd8b1f297583636bb6 P 8315d22c56f7439a9e1fc82f153f018c [term 1 LEADER]: Received request to transfer leadership
I20240501 07:59:15.155596 9056 consensus_queue.cc:237] T 12ce6a21d06c4ba889542758be9dc62d P a505e7cb133c4392bfda8a4a20874ae9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { replace: true } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: true } }
I20240501 07:59:15.156951 8925 raft_consensus.cc:1261] T 12ce6a21d06c4ba889542758be9dc62d P 13d05ebdca8c4f489c597ce550dd773d [term 1 FOLLOWER]: Refusing update from remote peer a505e7cb133c4392bfda8a4a20874ae9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:15.157056 8793 raft_consensus.cc:1261] T 12ce6a21d06c4ba889542758be9dc62d P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Refusing update from remote peer a505e7cb133c4392bfda8a4a20874ae9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 07:59:15.157148 8992 consensus_peers.cc:485] T 12ce6a21d06c4ba889542758be9dc62d P a505e7cb133c4392bfda8a4a20874ae9 -> Peer 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131:34471): Couldn't send request to peer 3b164cbbc00f4e8eb3254ddd349a3ff9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 12ce6a21d06c4ba889542758be9dc62d. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:15.157465 12488 consensus_queue.cc:1035] T 12ce6a21d06c4ba889542758be9dc62d P a505e7cb133c4392bfda8a4a20874ae9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 07:59:15.157675 12608 consensus_queue.cc:1035] T 12ce6a21d06c4ba889542758be9dc62d P a505e7cb133c4392bfda8a4a20874ae9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 07:59:15.159453 11453 raft_consensus.cc:2932] T 12ce6a21d06c4ba889542758be9dc62d P a505e7cb133c4392bfda8a4a20874ae9 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { replace: true } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: true } } }
I20240501 07:59:15.159883 8925 raft_consensus.cc:2932] T 12ce6a21d06c4ba889542758be9dc62d P 13d05ebdca8c4f489c597ce550dd773d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { replace: true } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: true } } }
I20240501 07:59:15.161183 8793 raft_consensus.cc:2932] T 12ce6a21d06c4ba889542758be9dc62d P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { replace: true } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: true } } }
I20240501 07:59:15.163719 6171 catalog_manager.cc:5484] T 12ce6a21d06c4ba889542758be9dc62d P 13d05ebdca8c4f489c597ce550dd773d reported cstate change: config changed from index -1 to 2, NON_VOTER 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131) added. New cstate: current_term: 1 leader_uuid: "a505e7cb133c4392bfda8a4a20874ae9" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { replace: true } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: true } } }
I20240501 07:59:15.177935 12019 raft_consensus.cc:1050] T 542c9ac08f7d4bec9b15a309c2e21364 P 13d05ebdca8c4f489c597ce550dd773d: attempting to promote NON_VOTER 3b164cbbc00f4e8eb3254ddd349a3ff9 to VOTER
I20240501 07:59:15.178349 12019 consensus_queue.cc:237] T 542c9ac08f7d4bec9b15a309c2e21364 P 13d05ebdca8c4f489c597ce550dd773d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { replace: true } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: false } }
I20240501 07:59:15.179944 6707 raft_consensus.cc:1261] T 542c9ac08f7d4bec9b15a309c2e21364 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Refusing update from remote peer 13d05ebdca8c4f489c597ce550dd773d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 07:59:15.180011 9056 raft_consensus.cc:1261] T 542c9ac08f7d4bec9b15a309c2e21364 P a505e7cb133c4392bfda8a4a20874ae9 [term 1 FOLLOWER]: Refusing update from remote peer 13d05ebdca8c4f489c597ce550dd773d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 07:59:15.180035 6577 raft_consensus.cc:1261] T 542c9ac08f7d4bec9b15a309c2e21364 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 LEARNER]: Refusing update from remote peer 13d05ebdca8c4f489c597ce550dd773d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 07:59:15.180325 9400 consensus_queue.cc:1035] T 542c9ac08f7d4bec9b15a309c2e21364 P 13d05ebdca8c4f489c597ce550dd773d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 07:59:15.180503 10478 consensus_queue.cc:1035] T 542c9ac08f7d4bec9b15a309c2e21364 P 13d05ebdca8c4f489c597ce550dd773d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 07:59:15.180938 10478 consensus_queue.cc:1035] T 542c9ac08f7d4bec9b15a309c2e21364 P 13d05ebdca8c4f489c597ce550dd773d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 07:59:15.184650 11215 raft_consensus.cc:2932] T 542c9ac08f7d4bec9b15a309c2e21364 P 13d05ebdca8c4f489c597ce550dd773d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { replace: true } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: false } } }
I20240501 07:59:15.185380 9056 raft_consensus.cc:2932] T 542c9ac08f7d4bec9b15a309c2e21364 P a505e7cb133c4392bfda8a4a20874ae9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { replace: true } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: false } } }
I20240501 07:59:15.185392 6707 raft_consensus.cc:2932] T 542c9ac08f7d4bec9b15a309c2e21364 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { replace: true } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: false } } }
I20240501 07:59:15.188647 6170 catalog_manager.cc:5484] T 542c9ac08f7d4bec9b15a309c2e21364 P 13d05ebdca8c4f489c597ce550dd773d reported cstate change: config changed from index 2 to 3, 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "13d05ebdca8c4f489c597ce550dd773d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 07:59:15.192375 6577 raft_consensus.cc:2932] T 542c9ac08f7d4bec9b15a309c2e21364 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { replace: true } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: false } } }
I20240501 07:59:15.193187 12492 ts_tablet_manager.cc:902] T 59e171892fc1414fbcfd6a8c090854f3 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Initiating tablet copy from peer 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134:40141)
I20240501 07:59:15.193722 12492 tablet_copy_client.cc:308] T 59e171892fc1414fbcfd6a8c090854f3 P 3b164cbbc00f4e8eb3254ddd349a3ff9: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.134:40141
I20240501 07:59:15.194566 8945 tablet_copy_service.cc:140] P 13d05ebdca8c4f489c597ce550dd773d: Received BeginTabletCopySession request for tablet 59e171892fc1414fbcfd6a8c090854f3 from peer 3b164cbbc00f4e8eb3254ddd349a3ff9 ({username='slave'} at 127.0.99.131:60487)
I20240501 07:59:15.194676 8945 tablet_copy_service.cc:161] P 13d05ebdca8c4f489c597ce550dd773d: Beginning new tablet copy session on tablet 59e171892fc1414fbcfd6a8c090854f3 from peer 3b164cbbc00f4e8eb3254ddd349a3ff9 at {username='slave'} at 127.0.99.131:60487: session id = 3b164cbbc00f4e8eb3254ddd349a3ff9-59e171892fc1414fbcfd6a8c090854f3
I20240501 07:59:15.195631 8945 tablet_copy_source_session.cc:205] T 59e171892fc1414fbcfd6a8c090854f3 P 13d05ebdca8c4f489c597ce550dd773d: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:15.196568 12492 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59e171892fc1414fbcfd6a8c090854f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:15.199020 12492 tablet_copy_client.cc:783] T 59e171892fc1414fbcfd6a8c090854f3 P 3b164cbbc00f4e8eb3254ddd349a3ff9: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:15.199064 8925 consensus_queue.cc:237] T 542c9ac08f7d4bec9b15a309c2e21364 P 13d05ebdca8c4f489c597ce550dd773d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: false } }
I20240501 07:59:15.199682 9189 raft_consensus.cc:1261] T ff9dc384dbac47dc9795617c64688943 P 83f20e84af4c40d09a4804b6b26e697f [term 3 FOLLOWER]: Refusing update from remote peer 13d05ebdca8c4f489c597ce550dd773d: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 3 index: 10. (index mismatch)
I20240501 07:59:15.200285 6577 raft_consensus.cc:1261] T 542c9ac08f7d4bec9b15a309c2e21364 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Refusing update from remote peer 13d05ebdca8c4f489c597ce550dd773d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:59:15.200645 11215 consensus_queue.cc:1035] T ff9dc384dbac47dc9795617c64688943 P 13d05ebdca8c4f489c597ce550dd773d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 07:59:15.200678 10478 consensus_queue.cc:1035] T 542c9ac08f7d4bec9b15a309c2e21364 P 13d05ebdca8c4f489c597ce550dd773d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:15.200891 8793 raft_consensus.cc:1203] T 114b6dbafa1c4c44ac42f1d2293e3125 P 74077b61d6f54342bf9736b4a9f2eade [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240501 07:59:15.201154 12492 tablet_copy_client.cc:647] T 59e171892fc1414fbcfd6a8c090854f3 P 3b164cbbc00f4e8eb3254ddd349a3ff9: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:15.203454 12492 tablet_copy_client.cc:523] T 59e171892fc1414fbcfd6a8c090854f3 P 3b164cbbc00f4e8eb3254ddd349a3ff9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:15.203521 6313 raft_consensus.cc:1261] T ff9dc384dbac47dc9795617c64688943 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 3 FOLLOWER]: Refusing update from remote peer 13d05ebdca8c4f489c597ce550dd773d: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 3 index: 10. (index mismatch)
I20240501 07:59:15.203958 6707 raft_consensus.cc:1261] T 542c9ac08f7d4bec9b15a309c2e21364 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Refusing update from remote peer 13d05ebdca8c4f489c597ce550dd773d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:59:15.204236 10478 consensus_queue.cc:1035] T ff9dc384dbac47dc9795617c64688943 P 13d05ebdca8c4f489c597ce550dd773d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 07:59:15.205384 10478 consensus_queue.cc:1035] T 542c9ac08f7d4bec9b15a309c2e21364 P 13d05ebdca8c4f489c597ce550dd773d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:15.206027 12492 tablet_bootstrap.cc:492] T 59e171892fc1414fbcfd6a8c090854f3 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Bootstrap starting.
I20240501 07:59:15.207340 6707 raft_consensus.cc:1261] T ff9dc384dbac47dc9795617c64688943 P 1f9cb72be98941b5a8e504cbe34132b9 [term 3 FOLLOWER]: Refusing update from remote peer 13d05ebdca8c4f489c597ce550dd773d: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 3 index: 10. (index mismatch)
I20240501 07:59:15.207928 12019 raft_consensus.cc:2932] T 542c9ac08f7d4bec9b15a309c2e21364 P 13d05ebdca8c4f489c597ce550dd773d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: false } } }
I20240501 07:59:15.209046 11215 consensus_queue.cc:1035] T ff9dc384dbac47dc9795617c64688943 P 13d05ebdca8c4f489c597ce550dd773d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 07:59:15.210565 12492 tablet_bootstrap.cc:492] T 59e171892fc1414fbcfd6a8c090854f3 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:15.210824 12492 tablet_bootstrap.cc:492] T 59e171892fc1414fbcfd6a8c090854f3 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Bootstrap complete.
I20240501 07:59:15.210989 12492 ts_tablet_manager.cc:1367] T 59e171892fc1414fbcfd6a8c090854f3 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20240501 07:59:15.211484 6159 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 542c9ac08f7d4bec9b15a309c2e21364 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 07:59:15.211437 12492 raft_consensus.cc:348] T 59e171892fc1414fbcfd6a8c090854f3 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { replace: true } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: true } }
I20240501 07:59:15.211556 12492 raft_consensus.cc:729] T 59e171892fc1414fbcfd6a8c090854f3 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3b164cbbc00f4e8eb3254ddd349a3ff9, State: Initialized, Role: LEARNER
I20240501 07:59:15.211658 12492 consensus_queue.cc:260] T 59e171892fc1414fbcfd6a8c090854f3 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { replace: true } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: true } }
I20240501 07:59:15.211995 12492 ts_tablet_manager.cc:1398] T 59e171892fc1414fbcfd6a8c090854f3 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240501 07:59:15.212013 8925 consensus_queue.cc:237] T ff9dc384dbac47dc9795617c64688943 P 13d05ebdca8c4f489c597ce550dd773d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 3.10, Last appended by leader: 9, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false } }
I20240501 07:59:15.213220 9189 raft_consensus.cc:1261] T ff9dc384dbac47dc9795617c64688943 P 83f20e84af4c40d09a4804b6b26e697f [term 3 FOLLOWER]: Refusing update from remote peer 13d05ebdca8c4f489c597ce550dd773d: Log matching property violated. Preceding OpId in replica: term: 3 index: 10. Preceding OpId from leader: term: 3 index: 11. (index mismatch)
I20240501 07:59:15.213706 11215 consensus_queue.cc:1035] T ff9dc384dbac47dc9795617c64688943 P 13d05ebdca8c4f489c597ce550dd773d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 07:59:15.216060 8945 tablet_copy_service.cc:342] P 13d05ebdca8c4f489c597ce550dd773d: Request end of tablet copy session 3b164cbbc00f4e8eb3254ddd349a3ff9-59e171892fc1414fbcfd6a8c090854f3 received from {username='slave'} at 127.0.99.131:60487
I20240501 07:59:15.216148 8945 tablet_copy_service.cc:433] P 13d05ebdca8c4f489c597ce550dd773d: ending tablet copy session 3b164cbbc00f4e8eb3254ddd349a3ff9-59e171892fc1414fbcfd6a8c090854f3 on tablet 59e171892fc1414fbcfd6a8c090854f3 with peer 3b164cbbc00f4e8eb3254ddd349a3ff9
I20240501 07:59:15.216440 6313 raft_consensus.cc:1261] T ff9dc384dbac47dc9795617c64688943 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 3 FOLLOWER]: Refusing update from remote peer 13d05ebdca8c4f489c597ce550dd773d: Log matching property violated. Preceding OpId in replica: term: 3 index: 10. Preceding OpId from leader: term: 3 index: 11. (index mismatch)
I20240501 07:59:15.217141 6707 raft_consensus.cc:2932] T 542c9ac08f7d4bec9b15a309c2e21364 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: false } } }
I20240501 07:59:15.217363 12019 consensus_queue.cc:1035] T bab84387a40d40c3b8af7f09eb462886 P 13d05ebdca8c4f489c597ce550dd773d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:15.219089 6170 catalog_manager.cc:5484] T 542c9ac08f7d4bec9b15a309c2e21364 P 13d05ebdca8c4f489c597ce550dd773d reported cstate change: config changed from index 3 to 4, VOTER a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135) evicted. New cstate: current_term: 1 leader_uuid: "13d05ebdca8c4f489c597ce550dd773d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 07:59:15.221537 6171 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55950:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 07:59:15.222146 10478 consensus_queue.cc:1035] T ff9dc384dbac47dc9795617c64688943 P 13d05ebdca8c4f489c597ce550dd773d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 07:59:15.222947 9035 tablet_service.cc:1514] Processing DeleteTablet for tablet 542c9ac08f7d4bec9b15a309c2e21364 with delete_type TABLET_DATA_TOMBSTONED (TS a505e7cb133c4392bfda8a4a20874ae9 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:44236
I20240501 07:59:15.223269 12447 tablet_replica.cc:330] T 542c9ac08f7d4bec9b15a309c2e21364 P a505e7cb133c4392bfda8a4a20874ae9: stopping tablet replica
I20240501 07:59:15.223415 12447 raft_consensus.cc:2229] T 542c9ac08f7d4bec9b15a309c2e21364 P a505e7cb133c4392bfda8a4a20874ae9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:15.223539 12447 raft_consensus.cc:2258] T 542c9ac08f7d4bec9b15a309c2e21364 P a505e7cb133c4392bfda8a4a20874ae9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:15.225245 12447 ts_tablet_manager.cc:1875] T 542c9ac08f7d4bec9b15a309c2e21364 P a505e7cb133c4392bfda8a4a20874ae9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:59:15.227710 12447 ts_tablet_manager.cc:1888] T 542c9ac08f7d4bec9b15a309c2e21364 P a505e7cb133c4392bfda8a4a20874ae9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 07:59:15.227787 12447 log.cc:1196] T 542c9ac08f7d4bec9b15a309c2e21364 P a505e7cb133c4392bfda8a4a20874ae9: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/542c9ac08f7d4bec9b15a309c2e21364
I20240501 07:59:15.228340 6158 catalog_manager.cc:4830] TS a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): tablet 542c9ac08f7d4bec9b15a309c2e21364 (table rebalancer_extra_table_8 [id=7f1dc10bb484406aa39a54213d4854b9]) successfully deleted
I20240501 07:59:15.229120 12019 raft_consensus.cc:2932] T ff9dc384dbac47dc9795617c64688943 P 13d05ebdca8c4f489c597ce550dd773d [term 3 LEADER]: Committing config change with OpId 3.11: config changed from index 9 to 11, VOTER 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false } } }
I20240501 07:59:15.230109 12574 tablet.cc:1722] T f2ce44c1d8c44b9789b21af5e889a419 P 83f20e84af4c40d09a4804b6b26e697f: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 07:59:15.230456 12574 tablet.cc:1620] T f2ce44c1d8c44b9789b21af5e889a419 P 83f20e84af4c40d09a4804b6b26e697f: MemRowSet was empty: no flush needed.
I20240501 07:59:15.231158 6577 raft_consensus.cc:2932] T 542c9ac08f7d4bec9b15a309c2e21364 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: false } } }
I20240501 07:59:15.233062 12576 tablet.cc:1722] T f2ce44c1d8c44b9789b21af5e889a419 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 07:59:15.233379 12576 tablet.cc:1620] T f2ce44c1d8c44b9789b21af5e889a419 P 3b164cbbc00f4e8eb3254ddd349a3ff9: MemRowSet was empty: no flush needed.
I20240501 07:59:15.233824 9189 raft_consensus.cc:2932] T ff9dc384dbac47dc9795617c64688943 P 83f20e84af4c40d09a4804b6b26e697f [term 3 FOLLOWER]: Committing config change with OpId 3.11: config changed from index 9 to 11, VOTER 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false } } }
I20240501 07:59:15.235345 6159 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ff9dc384dbac47dc9795617c64688943 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 07:59:15.237906 10478 consensus_queue.cc:1035] T bab84387a40d40c3b8af7f09eb462886 P 13d05ebdca8c4f489c597ce550dd773d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:15.238075 6313 raft_consensus.cc:2932] T ff9dc384dbac47dc9795617c64688943 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 3 FOLLOWER]: Committing config change with OpId 3.11: config changed from index 9 to 11, VOTER 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false } } }
I20240501 07:59:15.238205 6166 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55950:
name: "rebalancer_extra_table_20"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 07:59:15.239552 12575 tablet.cc:1722] T f2ce44c1d8c44b9789b21af5e889a419 P 74077b61d6f54342bf9736b4a9f2eade: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 07:59:15.239907 12575 tablet.cc:1620] T f2ce44c1d8c44b9789b21af5e889a419 P 74077b61d6f54342bf9736b4a9f2eade: MemRowSet was empty: no flush needed.
I20240501 07:59:15.240983 6171 catalog_manager.cc:5484] T ff9dc384dbac47dc9795617c64688943 P 13d05ebdca8c4f489c597ce550dd773d reported cstate change: config changed from index 9 to 11, VOTER 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132) evicted. New cstate: current_term: 3 leader_uuid: "13d05ebdca8c4f489c597ce550dd773d" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 07:59:15.244362 6687 tablet_service.cc:1514] Processing DeleteTablet for tablet ff9dc384dbac47dc9795617c64688943 with delete_type TABLET_DATA_TOMBSTONED (TS 1f9cb72be98941b5a8e504cbe34132b9 not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:38524
I20240501 07:59:15.245024 6172 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 0bc706e85ce74cef97a4827ea4d4aa65: rebalancer_dynamic_table [id=956c08c49bed4f7ea842f14bdfec9e03] alter complete (version 2)
I20240501 07:59:15.250355 12654 tablet_replica.cc:330] T ff9dc384dbac47dc9795617c64688943 P 1f9cb72be98941b5a8e504cbe34132b9: stopping tablet replica
I20240501 07:59:15.250543 12654 raft_consensus.cc:2229] T ff9dc384dbac47dc9795617c64688943 P 1f9cb72be98941b5a8e504cbe34132b9 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:15.250679 12654 raft_consensus.cc:2258] T ff9dc384dbac47dc9795617c64688943 P 1f9cb72be98941b5a8e504cbe34132b9 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:15.253230 12654 ts_tablet_manager.cc:1875] T ff9dc384dbac47dc9795617c64688943 P 1f9cb72be98941b5a8e504cbe34132b9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:59:15.253798 9035 tablet_service.cc:1467] Processing CreateTablet for tablet 8f36eefcda3f475d9571e92fdeb52eb4 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=315c2b77bb314ca393ceabc7126b3af7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:15.254153 9035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f36eefcda3f475d9571e92fdeb52eb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:15.254462 8774 tablet_service.cc:1467] Processing CreateTablet for tablet a93ce5841c7f40a9ab23f88efe7b85bb (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=315c2b77bb314ca393ceabc7126b3af7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:15.254765 8774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a93ce5841c7f40a9ab23f88efe7b85bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:15.255890 12654 ts_tablet_manager.cc:1888] T ff9dc384dbac47dc9795617c64688943 P 1f9cb72be98941b5a8e504cbe34132b9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.10
I20240501 07:59:15.255966 12654 log.cc:1196] T ff9dc384dbac47dc9795617c64688943 P 1f9cb72be98941b5a8e504cbe34132b9: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/ff9dc384dbac47dc9795617c64688943
I20240501 07:59:15.257194 12283 tablet_bootstrap.cc:492] T a93ce5841c7f40a9ab23f88efe7b85bb P 74077b61d6f54342bf9736b4a9f2eade: Bootstrap starting.
I20240501 07:59:15.257246 6424 tablet_service.cc:1467] Processing CreateTablet for tablet 8f36eefcda3f475d9571e92fdeb52eb4 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=315c2b77bb314ca393ceabc7126b3af7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:15.257241 6687 tablet_service.cc:1467] Processing CreateTablet for tablet 4bc915ffbbf345cbbe1195f8dd3ab874 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=315c2b77bb314ca393ceabc7126b3af7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:15.257511 6687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4bc915ffbbf345cbbe1195f8dd3ab874. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:15.257516 6424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f36eefcda3f475d9571e92fdeb52eb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:15.256894 6159 catalog_manager.cc:4830] TS 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132:36773): tablet ff9dc384dbac47dc9795617c64688943 (table rebalance_test_table_1 [id=d0e1f982440849beb494081bea9cb211]) successfully deleted
I20240501 07:59:15.258231 12283 tablet_bootstrap.cc:654] T a93ce5841c7f40a9ab23f88efe7b85bb P 74077b61d6f54342bf9736b4a9f2eade: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:15.258494 9035 tablet_service.cc:1467] Processing CreateTablet for tablet a93ce5841c7f40a9ab23f88efe7b85bb (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=315c2b77bb314ca393ceabc7126b3af7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:15.258755 9035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a93ce5841c7f40a9ab23f88efe7b85bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:15.259951 8905 tablet_service.cc:1467] Processing CreateTablet for tablet 4bc915ffbbf345cbbe1195f8dd3ab874 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=315c2b77bb314ca393ceabc7126b3af7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:15.260072 8904 tablet_service.cc:1467] Processing CreateTablet for tablet 8f36eefcda3f475d9571e92fdeb52eb4 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=315c2b77bb314ca393ceabc7126b3af7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:15.260218 8905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4bc915ffbbf345cbbe1195f8dd3ab874. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:15.260622 8904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f36eefcda3f475d9571e92fdeb52eb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:15.261363 12283 tablet_bootstrap.cc:492] T a93ce5841c7f40a9ab23f88efe7b85bb P 74077b61d6f54342bf9736b4a9f2eade: No bootstrap required, opened a new log
I20240501 07:59:15.261451 12283 ts_tablet_manager.cc:1367] T a93ce5841c7f40a9ab23f88efe7b85bb P 74077b61d6f54342bf9736b4a9f2eade: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240501 07:59:15.261881 12655 tablet_bootstrap.cc:492] T 8f36eefcda3f475d9571e92fdeb52eb4 P a505e7cb133c4392bfda8a4a20874ae9: Bootstrap starting.
I20240501 07:59:15.262003 12283 raft_consensus.cc:348] T a93ce5841c7f40a9ab23f88efe7b85bb P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } }
I20240501 07:59:15.262111 12283 raft_consensus.cc:374] T a93ce5841c7f40a9ab23f88efe7b85bb P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:15.262142 12283 raft_consensus.cc:729] T a93ce5841c7f40a9ab23f88efe7b85bb P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74077b61d6f54342bf9736b4a9f2eade, State: Initialized, Role: FOLLOWER
I20240501 07:59:15.262254 12283 consensus_queue.cc:260] T a93ce5841c7f40a9ab23f88efe7b85bb P 74077b61d6f54342bf9736b4a9f2eade [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } }
I20240501 07:59:15.262567 12283 ts_tablet_manager.cc:1398] T a93ce5841c7f40a9ab23f88efe7b85bb P 74077b61d6f54342bf9736b4a9f2eade: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 07:59:15.262960 12655 tablet_bootstrap.cc:654] T 8f36eefcda3f475d9571e92fdeb52eb4 P a505e7cb133c4392bfda8a4a20874ae9: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:15.264097 12658 tablet_bootstrap.cc:492] T 8f36eefcda3f475d9571e92fdeb52eb4 P 8315d22c56f7439a9e1fc82f153f018c: Bootstrap starting.
I20240501 07:59:15.265085 12658 tablet_bootstrap.cc:654] T 8f36eefcda3f475d9571e92fdeb52eb4 P 8315d22c56f7439a9e1fc82f153f018c: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:15.265420 9169 tablet_service.cc:1467] Processing CreateTablet for tablet a93ce5841c7f40a9ab23f88efe7b85bb (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=315c2b77bb314ca393ceabc7126b3af7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:15.265709 9169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a93ce5841c7f40a9ab23f88efe7b85bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:15.267623 12280 tablet_bootstrap.cc:492] T a93ce5841c7f40a9ab23f88efe7b85bb P 83f20e84af4c40d09a4804b6b26e697f: Bootstrap starting.
I20240501 07:59:15.267796 9167 tablet_service.cc:1467] Processing CreateTablet for tablet 4bc915ffbbf345cbbe1195f8dd3ab874 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=315c2b77bb314ca393ceabc7126b3af7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:15.268025 9167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4bc915ffbbf345cbbe1195f8dd3ab874. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:15.268541 12280 tablet_bootstrap.cc:654] T a93ce5841c7f40a9ab23f88efe7b85bb P 83f20e84af4c40d09a4804b6b26e697f: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:15.270227 12280 tablet_bootstrap.cc:492] T a93ce5841c7f40a9ab23f88efe7b85bb P 83f20e84af4c40d09a4804b6b26e697f: No bootstrap required, opened a new log
I20240501 07:59:15.270310 12280 ts_tablet_manager.cc:1367] T a93ce5841c7f40a9ab23f88efe7b85bb P 83f20e84af4c40d09a4804b6b26e697f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 07:59:15.270722 12280 raft_consensus.cc:348] T a93ce5841c7f40a9ab23f88efe7b85bb P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } }
I20240501 07:59:15.270819 12280 raft_consensus.cc:374] T a93ce5841c7f40a9ab23f88efe7b85bb P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:15.270848 12280 raft_consensus.cc:729] T a93ce5841c7f40a9ab23f88efe7b85bb P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83f20e84af4c40d09a4804b6b26e697f, State: Initialized, Role: FOLLOWER
I20240501 07:59:15.270960 12280 consensus_queue.cc:260] T a93ce5841c7f40a9ab23f88efe7b85bb P 83f20e84af4c40d09a4804b6b26e697f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } }
I20240501 07:59:15.271205 12280 ts_tablet_manager.cc:1398] T a93ce5841c7f40a9ab23f88efe7b85bb P 83f20e84af4c40d09a4804b6b26e697f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:15.271353 12280 tablet_bootstrap.cc:492] T 4bc915ffbbf345cbbe1195f8dd3ab874 P 83f20e84af4c40d09a4804b6b26e697f: Bootstrap starting.
I20240501 07:59:15.272213 12280 tablet_bootstrap.cc:654] T 4bc915ffbbf345cbbe1195f8dd3ab874 P 83f20e84af4c40d09a4804b6b26e697f: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:15.274246 12280 tablet_bootstrap.cc:492] T 4bc915ffbbf345cbbe1195f8dd3ab874 P 83f20e84af4c40d09a4804b6b26e697f: No bootstrap required, opened a new log
I20240501 07:59:15.274343 12280 ts_tablet_manager.cc:1367] T 4bc915ffbbf345cbbe1195f8dd3ab874 P 83f20e84af4c40d09a4804b6b26e697f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 07:59:15.275194 12657 tablet_bootstrap.cc:492] T 4bc915ffbbf345cbbe1195f8dd3ab874 P 1f9cb72be98941b5a8e504cbe34132b9: Bootstrap starting.
I20240501 07:59:15.275344 12280 raft_consensus.cc:348] T 4bc915ffbbf345cbbe1195f8dd3ab874 P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } }
I20240501 07:59:15.275454 12280 raft_consensus.cc:374] T 4bc915ffbbf345cbbe1195f8dd3ab874 P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:15.275530 12280 raft_consensus.cc:729] T 4bc915ffbbf345cbbe1195f8dd3ab874 P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83f20e84af4c40d09a4804b6b26e697f, State: Initialized, Role: FOLLOWER
I20240501 07:59:15.275666 12280 consensus_queue.cc:260] T 4bc915ffbbf345cbbe1195f8dd3ab874 P 83f20e84af4c40d09a4804b6b26e697f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } }
I20240501 07:59:15.275856 12659 tablet_bootstrap.cc:492] T 4bc915ffbbf345cbbe1195f8dd3ab874 P 13d05ebdca8c4f489c597ce550dd773d: Bootstrap starting.
I20240501 07:59:15.276005 12280 ts_tablet_manager.cc:1398] T 4bc915ffbbf345cbbe1195f8dd3ab874 P 83f20e84af4c40d09a4804b6b26e697f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:59:15.277210 12659 tablet_bootstrap.cc:654] T 4bc915ffbbf345cbbe1195f8dd3ab874 P 13d05ebdca8c4f489c597ce550dd773d: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:15.277949 12657 tablet_bootstrap.cc:654] T 4bc915ffbbf345cbbe1195f8dd3ab874 P 1f9cb72be98941b5a8e504cbe34132b9: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:15.279525 12655 tablet_bootstrap.cc:492] T 8f36eefcda3f475d9571e92fdeb52eb4 P a505e7cb133c4392bfda8a4a20874ae9: No bootstrap required, opened a new log
I20240501 07:59:15.279618 12655 ts_tablet_manager.cc:1367] T 8f36eefcda3f475d9571e92fdeb52eb4 P a505e7cb133c4392bfda8a4a20874ae9: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.000s
I20240501 07:59:15.280086 12655 raft_consensus.cc:348] T 8f36eefcda3f475d9571e92fdeb52eb4 P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } }
I20240501 07:59:15.280189 12655 raft_consensus.cc:374] T 8f36eefcda3f475d9571e92fdeb52eb4 P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:15.280216 12655 raft_consensus.cc:729] T 8f36eefcda3f475d9571e92fdeb52eb4 P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a505e7cb133c4392bfda8a4a20874ae9, State: Initialized, Role: FOLLOWER
I20240501 07:59:15.280333 12655 consensus_queue.cc:260] T 8f36eefcda3f475d9571e92fdeb52eb4 P a505e7cb133c4392bfda8a4a20874ae9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } }
I20240501 07:59:15.281194 12655 ts_tablet_manager.cc:1398] T 8f36eefcda3f475d9571e92fdeb52eb4 P a505e7cb133c4392bfda8a4a20874ae9: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 07:59:15.281350 12655 tablet_bootstrap.cc:492] T a93ce5841c7f40a9ab23f88efe7b85bb P a505e7cb133c4392bfda8a4a20874ae9: Bootstrap starting.
I20240501 07:59:15.282238 12655 tablet_bootstrap.cc:654] T a93ce5841c7f40a9ab23f88efe7b85bb P a505e7cb133c4392bfda8a4a20874ae9: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:15.282562 12659 tablet_bootstrap.cc:492] T 4bc915ffbbf345cbbe1195f8dd3ab874 P 13d05ebdca8c4f489c597ce550dd773d: No bootstrap required, opened a new log
I20240501 07:59:15.282649 12659 ts_tablet_manager.cc:1367] T 4bc915ffbbf345cbbe1195f8dd3ab874 P 13d05ebdca8c4f489c597ce550dd773d: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240501 07:59:15.283262 12655 tablet_bootstrap.cc:492] T a93ce5841c7f40a9ab23f88efe7b85bb P a505e7cb133c4392bfda8a4a20874ae9: No bootstrap required, opened a new log
I20240501 07:59:15.283198 12659 raft_consensus.cc:348] T 4bc915ffbbf345cbbe1195f8dd3ab874 P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } }
I20240501 07:59:15.283315 12659 raft_consensus.cc:374] T 4bc915ffbbf345cbbe1195f8dd3ab874 P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:15.283382 12659 raft_consensus.cc:729] T 4bc915ffbbf345cbbe1195f8dd3ab874 P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13d05ebdca8c4f489c597ce550dd773d, State: Initialized, Role: FOLLOWER
I20240501 07:59:15.283385 12655 ts_tablet_manager.cc:1367] T a93ce5841c7f40a9ab23f88efe7b85bb P a505e7cb133c4392bfda8a4a20874ae9: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240501 07:59:15.283526 12659 consensus_queue.cc:260] T 4bc915ffbbf345cbbe1195f8dd3ab874 P 13d05ebdca8c4f489c597ce550dd773d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } }
I20240501 07:59:15.283874 12659 ts_tablet_manager.cc:1398] T 4bc915ffbbf345cbbe1195f8dd3ab874 P 13d05ebdca8c4f489c597ce550dd773d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:15.284171 12659 tablet_bootstrap.cc:492] T 8f36eefcda3f475d9571e92fdeb52eb4 P 13d05ebdca8c4f489c597ce550dd773d: Bootstrap starting.
I20240501 07:59:15.284210 12657 tablet_bootstrap.cc:492] T 4bc915ffbbf345cbbe1195f8dd3ab874 P 1f9cb72be98941b5a8e504cbe34132b9: No bootstrap required, opened a new log
I20240501 07:59:15.284370 12657 ts_tablet_manager.cc:1367] T 4bc915ffbbf345cbbe1195f8dd3ab874 P 1f9cb72be98941b5a8e504cbe34132b9: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240501 07:59:15.285228 12659 tablet_bootstrap.cc:654] T 8f36eefcda3f475d9571e92fdeb52eb4 P 13d05ebdca8c4f489c597ce550dd773d: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:15.285214 12657 raft_consensus.cc:348] T 4bc915ffbbf345cbbe1195f8dd3ab874 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } }
I20240501 07:59:15.285388 12657 raft_consensus.cc:374] T 4bc915ffbbf345cbbe1195f8dd3ab874 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:15.285476 12657 raft_consensus.cc:729] T 4bc915ffbbf345cbbe1195f8dd3ab874 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f9cb72be98941b5a8e504cbe34132b9, State: Initialized, Role: FOLLOWER
I20240501 07:59:15.285686 12657 consensus_queue.cc:260] T 4bc915ffbbf345cbbe1195f8dd3ab874 P 1f9cb72be98941b5a8e504cbe34132b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } }
I20240501 07:59:15.287475 12657 ts_tablet_manager.cc:1398] T 4bc915ffbbf345cbbe1195f8dd3ab874 P 1f9cb72be98941b5a8e504cbe34132b9: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 07:59:15.287876 12655 raft_consensus.cc:348] T a93ce5841c7f40a9ab23f88efe7b85bb P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } }
I20240501 07:59:15.288071 12655 raft_consensus.cc:374] T a93ce5841c7f40a9ab23f88efe7b85bb P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:15.288120 12658 tablet_bootstrap.cc:492] T 8f36eefcda3f475d9571e92fdeb52eb4 P 8315d22c56f7439a9e1fc82f153f018c: No bootstrap required, opened a new log
I20240501 07:59:15.288158 12655 raft_consensus.cc:729] T a93ce5841c7f40a9ab23f88efe7b85bb P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a505e7cb133c4392bfda8a4a20874ae9, State: Initialized, Role: FOLLOWER
I20240501 07:59:15.288221 12658 ts_tablet_manager.cc:1367] T 8f36eefcda3f475d9571e92fdeb52eb4 P 8315d22c56f7439a9e1fc82f153f018c: Time spent bootstrapping tablet: real 0.024s user 0.000s sys 0.002s
I20240501 07:59:15.288287 12655 consensus_queue.cc:260] T a93ce5841c7f40a9ab23f88efe7b85bb P a505e7cb133c4392bfda8a4a20874ae9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } }
I20240501 07:59:15.288617 12658 raft_consensus.cc:348] T 8f36eefcda3f475d9571e92fdeb52eb4 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } }
I20240501 07:59:15.288707 12658 raft_consensus.cc:374] T 8f36eefcda3f475d9571e92fdeb52eb4 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:15.288733 12658 raft_consensus.cc:729] T 8f36eefcda3f475d9571e92fdeb52eb4 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8315d22c56f7439a9e1fc82f153f018c, State: Initialized, Role: FOLLOWER
I20240501 07:59:15.288851 12658 consensus_queue.cc:260] T 8f36eefcda3f475d9571e92fdeb52eb4 P 8315d22c56f7439a9e1fc82f153f018c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } }
I20240501 07:59:15.289208 12655 ts_tablet_manager.cc:1398] T a93ce5841c7f40a9ab23f88efe7b85bb P a505e7cb133c4392bfda8a4a20874ae9: Time spent starting tablet: real 0.006s user 0.000s sys 0.001s
I20240501 07:59:15.290720 12658 ts_tablet_manager.cc:1398] T 8f36eefcda3f475d9571e92fdeb52eb4 P 8315d22c56f7439a9e1fc82f153f018c: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:59:15.294103 12659 tablet_bootstrap.cc:492] T 8f36eefcda3f475d9571e92fdeb52eb4 P 13d05ebdca8c4f489c597ce550dd773d: No bootstrap required, opened a new log
I20240501 07:59:15.294203 12659 ts_tablet_manager.cc:1367] T 8f36eefcda3f475d9571e92fdeb52eb4 P 13d05ebdca8c4f489c597ce550dd773d: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240501 07:59:15.294670 12659 raft_consensus.cc:348] T 8f36eefcda3f475d9571e92fdeb52eb4 P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } }
I20240501 07:59:15.294776 12659 raft_consensus.cc:374] T 8f36eefcda3f475d9571e92fdeb52eb4 P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:15.294804 12659 raft_consensus.cc:729] T 8f36eefcda3f475d9571e92fdeb52eb4 P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13d05ebdca8c4f489c597ce550dd773d, State: Initialized, Role: FOLLOWER
I20240501 07:59:15.294929 12659 consensus_queue.cc:260] T 8f36eefcda3f475d9571e92fdeb52eb4 P 13d05ebdca8c4f489c597ce550dd773d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } }
I20240501 07:59:15.295200 12659 ts_tablet_manager.cc:1398] T 8f36eefcda3f475d9571e92fdeb52eb4 P 13d05ebdca8c4f489c597ce550dd773d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:15.310972 12488 raft_consensus.cc:482] T 8f36eefcda3f475d9571e92fdeb52eb4 P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:15.311126 12488 raft_consensus.cc:504] T 8f36eefcda3f475d9571e92fdeb52eb4 P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } }
I20240501 07:59:15.311631 12488 leader_election.cc:288] T 8f36eefcda3f475d9571e92fdeb52eb4 P a505e7cb133c4392bfda8a4a20874ae9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130:32941), 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134:40141)
I20240501 07:59:15.312013 6444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f36eefcda3f475d9571e92fdeb52eb4" candidate_uuid: "a505e7cb133c4392bfda8a4a20874ae9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8315d22c56f7439a9e1fc82f153f018c" is_pre_election: true
I20240501 07:59:15.312094 8925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f36eefcda3f475d9571e92fdeb52eb4" candidate_uuid: "a505e7cb133c4392bfda8a4a20874ae9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13d05ebdca8c4f489c597ce550dd773d" is_pre_election: true
I20240501 07:59:15.312160 6444 raft_consensus.cc:2454] T 8f36eefcda3f475d9571e92fdeb52eb4 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a505e7cb133c4392bfda8a4a20874ae9 in term 0.
I20240501 07:59:15.312217 8925 raft_consensus.cc:2454] T 8f36eefcda3f475d9571e92fdeb52eb4 P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a505e7cb133c4392bfda8a4a20874ae9 in term 0.
I20240501 07:59:15.312465 8991 leader_election.cc:302] T 8f36eefcda3f475d9571e92fdeb52eb4 P a505e7cb133c4392bfda8a4a20874ae9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8315d22c56f7439a9e1fc82f153f018c, a505e7cb133c4392bfda8a4a20874ae9; no voters:
I20240501 07:59:15.312659 12488 raft_consensus.cc:2781] T 8f36eefcda3f475d9571e92fdeb52eb4 P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:15.312737 12488 raft_consensus.cc:482] T 8f36eefcda3f475d9571e92fdeb52eb4 P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:15.312805 12488 raft_consensus.cc:3037] T 8f36eefcda3f475d9571e92fdeb52eb4 P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:15.313920 12488 raft_consensus.cc:504] T 8f36eefcda3f475d9571e92fdeb52eb4 P a505e7cb133c4392bfda8a4a20874ae9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } }
I20240501 07:59:15.314309 12488 leader_election.cc:288] T 8f36eefcda3f475d9571e92fdeb52eb4 P a505e7cb133c4392bfda8a4a20874ae9 [CANDIDATE]: Term 1 election: Requested vote from peers 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130:32941), 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134:40141)
I20240501 07:59:15.314800 6444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f36eefcda3f475d9571e92fdeb52eb4" candidate_uuid: "a505e7cb133c4392bfda8a4a20874ae9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8315d22c56f7439a9e1fc82f153f018c"
I20240501 07:59:15.314813 8925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f36eefcda3f475d9571e92fdeb52eb4" candidate_uuid: "a505e7cb133c4392bfda8a4a20874ae9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13d05ebdca8c4f489c597ce550dd773d"
I20240501 07:59:15.314929 6444 raft_consensus.cc:3037] T 8f36eefcda3f475d9571e92fdeb52eb4 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:15.314939 8925 raft_consensus.cc:3037] T 8f36eefcda3f475d9571e92fdeb52eb4 P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:15.315812 8925 raft_consensus.cc:2454] T 8f36eefcda3f475d9571e92fdeb52eb4 P 13d05ebdca8c4f489c597ce550dd773d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a505e7cb133c4392bfda8a4a20874ae9 in term 1.
I20240501 07:59:15.315822 6444 raft_consensus.cc:2454] T 8f36eefcda3f475d9571e92fdeb52eb4 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a505e7cb133c4392bfda8a4a20874ae9 in term 1.
I20240501 07:59:15.316087 8991 leader_election.cc:302] T 8f36eefcda3f475d9571e92fdeb52eb4 P a505e7cb133c4392bfda8a4a20874ae9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8315d22c56f7439a9e1fc82f153f018c, a505e7cb133c4392bfda8a4a20874ae9; no voters:
I20240501 07:59:15.316370 12488 raft_consensus.cc:2781] T 8f36eefcda3f475d9571e92fdeb52eb4 P a505e7cb133c4392bfda8a4a20874ae9 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:15.316464 12488 raft_consensus.cc:686] T 8f36eefcda3f475d9571e92fdeb52eb4 P a505e7cb133c4392bfda8a4a20874ae9 [term 1 LEADER]: Becoming Leader. State: Replica: a505e7cb133c4392bfda8a4a20874ae9, State: Running, Role: LEADER
I20240501 07:59:15.316615 12488 consensus_queue.cc:237] T 8f36eefcda3f475d9571e92fdeb52eb4 P a505e7cb133c4392bfda8a4a20874ae9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } }
I20240501 07:59:15.319008 6165 catalog_manager.cc:5484] T 8f36eefcda3f475d9571e92fdeb52eb4 P a505e7cb133c4392bfda8a4a20874ae9 reported cstate change: term changed from 0 to 1, leader changed from <none> to a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135). New cstate: current_term: 1 leader_uuid: "a505e7cb133c4392bfda8a4a20874ae9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } health_report { overall_health: UNKNOWN } } }
I20240501 07:59:15.319908 6444 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6dc929e2a6ae45bd8b1f297583636bb6"
dest_uuid: "8315d22c56f7439a9e1fc82f153f018c"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:59474
I20240501 07:59:15.320021 6444 raft_consensus.cc:595] T 6dc929e2a6ae45bd8b1f297583636bb6 P 8315d22c56f7439a9e1fc82f153f018c [term 1 LEADER]: Received request to transfer leadership
I20240501 07:59:15.340534 12649 raft_consensus.cc:482] T a93ce5841c7f40a9ab23f88efe7b85bb P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:15.340667 12649 raft_consensus.cc:504] T a93ce5841c7f40a9ab23f88efe7b85bb P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } }
I20240501 07:59:15.341166 12649 leader_election.cc:288] T a93ce5841c7f40a9ab23f88efe7b85bb P 83f20e84af4c40d09a4804b6b26e697f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133:39859), a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607)
I20240501 07:59:15.341539 8793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a93ce5841c7f40a9ab23f88efe7b85bb" candidate_uuid: "83f20e84af4c40d09a4804b6b26e697f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74077b61d6f54342bf9736b4a9f2eade" is_pre_election: true
I20240501 07:59:15.341534 6575 consensus_queue.cc:237] T b7d57b7fd5234f50afbc859bc997f3e1 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: true } }
I20240501 07:59:15.341689 8793 raft_consensus.cc:2454] T a93ce5841c7f40a9ab23f88efe7b85bb P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83f20e84af4c40d09a4804b6b26e697f in term 0.
I20240501 07:59:15.341683 9056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a93ce5841c7f40a9ab23f88efe7b85bb" candidate_uuid: "83f20e84af4c40d09a4804b6b26e697f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a505e7cb133c4392bfda8a4a20874ae9" is_pre_election: true
I20240501 07:59:15.341816 9056 raft_consensus.cc:2454] T a93ce5841c7f40a9ab23f88efe7b85bb P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83f20e84af4c40d09a4804b6b26e697f in term 0.
I20240501 07:59:15.341982 9122 leader_election.cc:302] T a93ce5841c7f40a9ab23f88efe7b85bb P 83f20e84af4c40d09a4804b6b26e697f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74077b61d6f54342bf9736b4a9f2eade, 83f20e84af4c40d09a4804b6b26e697f; no voters:
I20240501 07:59:15.342140 12649 raft_consensus.cc:2781] T a93ce5841c7f40a9ab23f88efe7b85bb P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:15.342186 12649 raft_consensus.cc:482] T a93ce5841c7f40a9ab23f88efe7b85bb P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:15.342211 12649 raft_consensus.cc:3037] T a93ce5841c7f40a9ab23f88efe7b85bb P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:15.343093 8793 raft_consensus.cc:1261] T b7d57b7fd5234f50afbc859bc997f3e1 P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Refusing update from remote peer 3b164cbbc00f4e8eb3254ddd349a3ff9: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 07:59:15.343340 12649 raft_consensus.cc:504] T a93ce5841c7f40a9ab23f88efe7b85bb P 83f20e84af4c40d09a4804b6b26e697f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } }
I20240501 07:59:15.343549 12119 consensus_queue.cc:1035] T b7d57b7fd5234f50afbc859bc997f3e1 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 07:59:15.343680 12649 leader_election.cc:288] T a93ce5841c7f40a9ab23f88efe7b85bb P 83f20e84af4c40d09a4804b6b26e697f [CANDIDATE]: Term 1 election: Requested vote from peers 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133:39859), a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607)
I20240501 07:59:15.343988 6313 raft_consensus.cc:1261] T b7d57b7fd5234f50afbc859bc997f3e1 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Refusing update from remote peer 3b164cbbc00f4e8eb3254ddd349a3ff9: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 07:59:15.344137 8793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a93ce5841c7f40a9ab23f88efe7b85bb" candidate_uuid: "83f20e84af4c40d09a4804b6b26e697f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74077b61d6f54342bf9736b4a9f2eade"
W20240501 07:59:15.344187 6511 consensus_peers.cc:485] T b7d57b7fd5234f50afbc859bc997f3e1 P 3b164cbbc00f4e8eb3254ddd349a3ff9 -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b7d57b7fd5234f50afbc859bc997f3e1. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:15.344251 8793 raft_consensus.cc:3037] T a93ce5841c7f40a9ab23f88efe7b85bb P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:15.344360 12119 consensus_queue.cc:1035] T b7d57b7fd5234f50afbc859bc997f3e1 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 07:59:15.344640 9056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a93ce5841c7f40a9ab23f88efe7b85bb" candidate_uuid: "83f20e84af4c40d09a4804b6b26e697f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a505e7cb133c4392bfda8a4a20874ae9"
I20240501 07:59:15.344761 9056 raft_consensus.cc:3037] T a93ce5841c7f40a9ab23f88efe7b85bb P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:15.345324 8793 raft_consensus.cc:2454] T a93ce5841c7f40a9ab23f88efe7b85bb P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83f20e84af4c40d09a4804b6b26e697f in term 1.
I20240501 07:59:15.345614 9122 leader_election.cc:302] T a93ce5841c7f40a9ab23f88efe7b85bb P 83f20e84af4c40d09a4804b6b26e697f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74077b61d6f54342bf9736b4a9f2eade, 83f20e84af4c40d09a4804b6b26e697f; no voters:
I20240501 07:59:15.345775 12649 raft_consensus.cc:2781] T a93ce5841c7f40a9ab23f88efe7b85bb P 83f20e84af4c40d09a4804b6b26e697f [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:15.345853 12649 raft_consensus.cc:686] T a93ce5841c7f40a9ab23f88efe7b85bb P 83f20e84af4c40d09a4804b6b26e697f [term 1 LEADER]: Becoming Leader. State: Replica: 83f20e84af4c40d09a4804b6b26e697f, State: Running, Role: LEADER
I20240501 07:59:15.345937 9056 raft_consensus.cc:2454] T a93ce5841c7f40a9ab23f88efe7b85bb P a505e7cb133c4392bfda8a4a20874ae9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83f20e84af4c40d09a4804b6b26e697f in term 1.
I20240501 07:59:15.345989 12649 consensus_queue.cc:237] T a93ce5841c7f40a9ab23f88efe7b85bb P 83f20e84af4c40d09a4804b6b26e697f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } }
I20240501 07:59:15.346508 11107 raft_consensus.cc:2932] T b7d57b7fd5234f50afbc859bc997f3e1 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: true } } }
I20240501 07:59:15.346820 6171 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55950:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 07:59:15.347021 8794 raft_consensus.cc:2932] T b7d57b7fd5234f50afbc859bc997f3e1 P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: true } } }
I20240501 07:59:15.348207 6165 catalog_manager.cc:5484] T a93ce5841c7f40a9ab23f88efe7b85bb P 83f20e84af4c40d09a4804b6b26e697f reported cstate change: term changed from 0 to 1, leader changed from <none> to 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136). New cstate: current_term: 1 leader_uuid: "83f20e84af4c40d09a4804b6b26e697f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } health_report { overall_health: UNKNOWN } } }
I20240501 07:59:15.350550 6166 catalog_manager.cc:5484] T b7d57b7fd5234f50afbc859bc997f3e1 P 74077b61d6f54342bf9736b4a9f2eade reported cstate change: config changed from index 6 to 7, NON_VOTER 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: true } } }
I20240501 07:59:15.351735 6575 raft_consensus.cc:1203] T 28eaf22342ff4b5394f958d0b659273c P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 07:59:15.354161 6313 raft_consensus.cc:2932] T b7d57b7fd5234f50afbc859bc997f3e1 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: true } } }
I20240501 07:59:15.357347 12574 tablet.cc:1722] T f2ce44c1d8c44b9789b21af5e889a419 P 83f20e84af4c40d09a4804b6b26e697f: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 07:59:15.358568 12576 tablet.cc:1722] T f2ce44c1d8c44b9789b21af5e889a419 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 07:59:15.363685 6165 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 0bc706e85ce74cef97a4827ea4d4aa65: rebalancer_dynamic_table_new_name [id=956c08c49bed4f7ea842f14bdfec9e03] alter complete (version 3)
I20240501 07:59:15.363916 12575 tablet.cc:1722] T f2ce44c1d8c44b9789b21af5e889a419 P 74077b61d6f54342bf9736b4a9f2eade: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 07:59:15.417805 10111 raft_consensus.cc:979] T 6dc929e2a6ae45bd8b1f297583636bb6 P 8315d22c56f7439a9e1fc82f153f018c: : Instructing follower 74077b61d6f54342bf9736b4a9f2eade to start an election
I20240501 07:59:15.417910 10111 raft_consensus.cc:1067] T 6dc929e2a6ae45bd8b1f297583636bb6 P 8315d22c56f7439a9e1fc82f153f018c [term 1 LEADER]: Signalling peer 74077b61d6f54342bf9736b4a9f2eade to start an election
I20240501 07:59:15.418365 8794 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6dc929e2a6ae45bd8b1f297583636bb6"
dest_uuid: "74077b61d6f54342bf9736b4a9f2eade"
from {username='slave'} at 127.0.99.130:51123
I20240501 07:59:15.418514 8794 raft_consensus.cc:482] T 6dc929e2a6ae45bd8b1f297583636bb6 P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:59:15.418582 8794 raft_consensus.cc:3037] T 6dc929e2a6ae45bd8b1f297583636bb6 P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:15.419656 8794 raft_consensus.cc:504] T 6dc929e2a6ae45bd8b1f297583636bb6 P 74077b61d6f54342bf9736b4a9f2eade [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false } }
I20240501 07:59:15.420107 8794 leader_election.cc:288] T 6dc929e2a6ae45bd8b1f297583636bb6 P 74077b61d6f54342bf9736b4a9f2eade [CANDIDATE]: Term 2 election: Requested vote from peers 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129:46875), 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130:32941), 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149)
I20240501 07:59:15.420404 6313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6dc929e2a6ae45bd8b1f297583636bb6" candidate_uuid: "74077b61d6f54342bf9736b4a9f2eade" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f"
I20240501 07:59:15.420503 9188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6dc929e2a6ae45bd8b1f297583636bb6" candidate_uuid: "74077b61d6f54342bf9736b4a9f2eade" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "83f20e84af4c40d09a4804b6b26e697f"
I20240501 07:59:15.420549 6313 raft_consensus.cc:3037] T 6dc929e2a6ae45bd8b1f297583636bb6 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:15.420647 9188 raft_consensus.cc:3037] T 6dc929e2a6ae45bd8b1f297583636bb6 P 83f20e84af4c40d09a4804b6b26e697f [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:15.421417 6445 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6dc929e2a6ae45bd8b1f297583636bb6"
dest_uuid: "8315d22c56f7439a9e1fc82f153f018c"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:59484
I20240501 07:59:15.421581 6445 raft_consensus.cc:595] T 6dc929e2a6ae45bd8b1f297583636bb6 P 8315d22c56f7439a9e1fc82f153f018c [term 1 LEADER]: Received request to transfer leadership
I20240501 07:59:15.421664 6313 raft_consensus.cc:2454] T 6dc929e2a6ae45bd8b1f297583636bb6 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74077b61d6f54342bf9736b4a9f2eade in term 2.
I20240501 07:59:15.421685 9188 raft_consensus.cc:2454] T 6dc929e2a6ae45bd8b1f297583636bb6 P 83f20e84af4c40d09a4804b6b26e697f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74077b61d6f54342bf9736b4a9f2eade in term 2.
I20240501 07:59:15.421974 6444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6dc929e2a6ae45bd8b1f297583636bb6" candidate_uuid: "74077b61d6f54342bf9736b4a9f2eade" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "8315d22c56f7439a9e1fc82f153f018c"
I20240501 07:59:15.422082 6444 raft_consensus.cc:3032] T 6dc929e2a6ae45bd8b1f297583636bb6 P 8315d22c56f7439a9e1fc82f153f018c [term 1 LEADER]: Stepping down as leader of term 1
I20240501 07:59:15.422063 8727 leader_election.cc:302] T 6dc929e2a6ae45bd8b1f297583636bb6 P 74077b61d6f54342bf9736b4a9f2eade [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 74077b61d6f54342bf9736b4a9f2eade, 83f20e84af4c40d09a4804b6b26e697f, 8b8b3ad3aa8f4053b2167dd550f2086f; no voters:
I20240501 07:59:15.422124 6444 raft_consensus.cc:729] T 6dc929e2a6ae45bd8b1f297583636bb6 P 8315d22c56f7439a9e1fc82f153f018c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8315d22c56f7439a9e1fc82f153f018c, State: Running, Role: LEADER
I20240501 07:59:15.422295 12123 raft_consensus.cc:2781] T 6dc929e2a6ae45bd8b1f297583636bb6 P 74077b61d6f54342bf9736b4a9f2eade [term 2 FOLLOWER]: Leader election won for term 2
I20240501 07:59:15.422281 6444 consensus_queue.cc:260] T 6dc929e2a6ae45bd8b1f297583636bb6 P 8315d22c56f7439a9e1fc82f153f018c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false } }
I20240501 07:59:15.422382 12123 raft_consensus.cc:686] T 6dc929e2a6ae45bd8b1f297583636bb6 P 74077b61d6f54342bf9736b4a9f2eade [term 2 LEADER]: Becoming Leader. State: Replica: 74077b61d6f54342bf9736b4a9f2eade, State: Running, Role: LEADER
I20240501 07:59:15.422502 6444 raft_consensus.cc:3037] T 6dc929e2a6ae45bd8b1f297583636bb6 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:15.422510 12123 consensus_queue.cc:237] T 6dc929e2a6ae45bd8b1f297583636bb6 P 74077b61d6f54342bf9736b4a9f2eade [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false } }
I20240501 07:59:15.423512 6444 raft_consensus.cc:2454] T 6dc929e2a6ae45bd8b1f297583636bb6 P 8315d22c56f7439a9e1fc82f153f018c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74077b61d6f54342bf9736b4a9f2eade in term 2.
I20240501 07:59:15.424999 6171 catalog_manager.cc:5484] T 6dc929e2a6ae45bd8b1f297583636bb6 P 74077b61d6f54342bf9736b4a9f2eade reported cstate change: term changed from 1 to 2, leader changed from 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130) to 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133). New cstate: current_term: 2 leader_uuid: "74077b61d6f54342bf9736b4a9f2eade" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 07:59:15.427102 6444 raft_consensus.cc:1203] T 2f9e82c3f1df4d1292e37e9cea07186e P 8315d22c56f7439a9e1fc82f153f018c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 07:59:15.432525 8925 raft_consensus.cc:1203] T 6c3c0d6fe41f4ae196ae29beb9b68125 P 13d05ebdca8c4f489c597ce550dd773d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240501 07:59:15.466859 6171 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55950:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 07:59:15.467047 6171 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55950:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 07:59:15.469461 6171 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 0bc706e85ce74cef97a4827ea4d4aa65: Sending DeleteTablet for 3 replicas of tablet f2ce44c1d8c44b9789b21af5e889a419
I20240501 07:59:15.470054 9167 tablet_service.cc:1514] Processing DeleteTablet for tablet f2ce44c1d8c44b9789b21af5e889a419 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:15 UTC) from {username='slave'} at 127.0.0.1:33594
I20240501 07:59:15.470043 6556 tablet_service.cc:1514] Processing DeleteTablet for tablet f2ce44c1d8c44b9789b21af5e889a419 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:15 UTC) from {username='slave'} at 127.0.0.1:53874
I20240501 07:59:15.470161 8774 tablet_service.cc:1514] Processing DeleteTablet for tablet f2ce44c1d8c44b9789b21af5e889a419 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:15 UTC) from {username='slave'} at 127.0.0.1:43136
I20240501 07:59:15.470232 12354 tablet_replica.cc:330] T f2ce44c1d8c44b9789b21af5e889a419 P 3b164cbbc00f4e8eb3254ddd349a3ff9: stopping tablet replica
I20240501 07:59:15.470232 12510 tablet_replica.cc:330] T f2ce44c1d8c44b9789b21af5e889a419 P 83f20e84af4c40d09a4804b6b26e697f: stopping tablet replica
I20240501 07:59:15.470350 12510 raft_consensus.cc:2229] T f2ce44c1d8c44b9789b21af5e889a419 P 83f20e84af4c40d09a4804b6b26e697f [term 1 LEADER]: Raft consensus shutting down.
I20240501 07:59:15.470353 12354 raft_consensus.cc:2229] T f2ce44c1d8c44b9789b21af5e889a419 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:15.470463 12354 raft_consensus.cc:2258] T f2ce44c1d8c44b9789b21af5e889a419 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:15.470511 12768 tablet_replica.cc:330] T f2ce44c1d8c44b9789b21af5e889a419 P 74077b61d6f54342bf9736b4a9f2eade: stopping tablet replica
I20240501 07:59:15.470553 12510 raft_consensus.cc:2258] T f2ce44c1d8c44b9789b21af5e889a419 P 83f20e84af4c40d09a4804b6b26e697f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:15.470672 12768 raft_consensus.cc:2229] T f2ce44c1d8c44b9789b21af5e889a419 P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:15.470841 12768 raft_consensus.cc:2258] T f2ce44c1d8c44b9789b21af5e889a419 P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:15.471231 12354 ts_tablet_manager.cc:1875] T f2ce44c1d8c44b9789b21af5e889a419 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:59:15.471376 12768 ts_tablet_manager.cc:1875] T f2ce44c1d8c44b9789b21af5e889a419 P 74077b61d6f54342bf9736b4a9f2eade: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:59:15.473119 12354 ts_tablet_manager.cc:1888] T f2ce44c1d8c44b9789b21af5e889a419 P 3b164cbbc00f4e8eb3254ddd349a3ff9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:59:15.473202 12354 log.cc:1196] T f2ce44c1d8c44b9789b21af5e889a419 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/f2ce44c1d8c44b9789b21af5e889a419
I20240501 07:59:15.473273 12510 ts_tablet_manager.cc:1875] T f2ce44c1d8c44b9789b21af5e889a419 P 83f20e84af4c40d09a4804b6b26e697f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:59:15.473477 12354 ts_tablet_manager.cc:1909] T f2ce44c1d8c44b9789b21af5e889a419 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Deleting consensus metadata
I20240501 07:59:15.473675 12768 ts_tablet_manager.cc:1888] T f2ce44c1d8c44b9789b21af5e889a419 P 74077b61d6f54342bf9736b4a9f2eade: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:59:15.473733 12768 log.cc:1196] T f2ce44c1d8c44b9789b21af5e889a419 P 74077b61d6f54342bf9736b4a9f2eade: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/f2ce44c1d8c44b9789b21af5e889a419
I20240501 07:59:15.474004 12768 ts_tablet_manager.cc:1909] T f2ce44c1d8c44b9789b21af5e889a419 P 74077b61d6f54342bf9736b4a9f2eade: Deleting consensus metadata
I20240501 07:59:15.474334 6160 catalog_manager.cc:4830] TS 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131:34471): tablet f2ce44c1d8c44b9789b21af5e889a419 (table rebalancer_dynamic_table_new_name [id=956c08c49bed4f7ea842f14bdfec9e03]) successfully deleted
I20240501 07:59:15.474653 6157 catalog_manager.cc:4830] TS 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133:39859): tablet f2ce44c1d8c44b9789b21af5e889a419 (table rebalancer_dynamic_table_new_name [id=956c08c49bed4f7ea842f14bdfec9e03]) successfully deleted
I20240501 07:59:15.475135 12510 ts_tablet_manager.cc:1888] T f2ce44c1d8c44b9789b21af5e889a419 P 83f20e84af4c40d09a4804b6b26e697f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:59:15.475195 12510 log.cc:1196] T f2ce44c1d8c44b9789b21af5e889a419 P 83f20e84af4c40d09a4804b6b26e697f: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/f2ce44c1d8c44b9789b21af5e889a419
I20240501 07:59:15.475382 12510 ts_tablet_manager.cc:1909] T f2ce44c1d8c44b9789b21af5e889a419 P 83f20e84af4c40d09a4804b6b26e697f: Deleting consensus metadata
I20240501 07:59:15.476217 6158 catalog_manager.cc:4830] TS 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): tablet f2ce44c1d8c44b9789b21af5e889a419 (table rebalancer_dynamic_table_new_name [id=956c08c49bed4f7ea842f14bdfec9e03]) successfully deleted
I20240501 07:59:15.495445 8925 raft_consensus.cc:1203] T a4727883b2544ed58b356f41dc8017a9 P 13d05ebdca8c4f489c597ce550dd773d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 07:59:15.497514 11402 raft_consensus.cc:482] T 4bc915ffbbf345cbbe1195f8dd3ab874 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:15.497642 11402 raft_consensus.cc:504] T 4bc915ffbbf345cbbe1195f8dd3ab874 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } }
I20240501 07:59:15.498107 11402 leader_election.cc:288] T 4bc915ffbbf345cbbe1195f8dd3ab874 P 1f9cb72be98941b5a8e504cbe34132b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149), 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134:40141)
I20240501 07:59:15.498451 9188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bc915ffbbf345cbbe1195f8dd3ab874" candidate_uuid: "1f9cb72be98941b5a8e504cbe34132b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83f20e84af4c40d09a4804b6b26e697f" is_pre_election: true
I20240501 07:59:15.498484 8925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bc915ffbbf345cbbe1195f8dd3ab874" candidate_uuid: "1f9cb72be98941b5a8e504cbe34132b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13d05ebdca8c4f489c597ce550dd773d" is_pre_election: true
I20240501 07:59:15.498589 9188 raft_consensus.cc:2454] T 4bc915ffbbf345cbbe1195f8dd3ab874 P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f9cb72be98941b5a8e504cbe34132b9 in term 0.
I20240501 07:59:15.498629 8925 raft_consensus.cc:2454] T 4bc915ffbbf345cbbe1195f8dd3ab874 P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f9cb72be98941b5a8e504cbe34132b9 in term 0.
I20240501 07:59:15.498905 6642 leader_election.cc:302] T 4bc915ffbbf345cbbe1195f8dd3ab874 P 1f9cb72be98941b5a8e504cbe34132b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f9cb72be98941b5a8e504cbe34132b9, 83f20e84af4c40d09a4804b6b26e697f; no voters:
I20240501 07:59:15.499179 11402 raft_consensus.cc:2781] T 4bc915ffbbf345cbbe1195f8dd3ab874 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:15.499236 11402 raft_consensus.cc:482] T 4bc915ffbbf345cbbe1195f8dd3ab874 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:15.499285 11402 raft_consensus.cc:3037] T 4bc915ffbbf345cbbe1195f8dd3ab874 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:15.500330 11402 raft_consensus.cc:504] T 4bc915ffbbf345cbbe1195f8dd3ab874 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } }
I20240501 07:59:15.500698 11402 leader_election.cc:288] T 4bc915ffbbf345cbbe1195f8dd3ab874 P 1f9cb72be98941b5a8e504cbe34132b9 [CANDIDATE]: Term 1 election: Requested vote from peers 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149), 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134:40141)
I20240501 07:59:15.501047 8925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bc915ffbbf345cbbe1195f8dd3ab874" candidate_uuid: "1f9cb72be98941b5a8e504cbe34132b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13d05ebdca8c4f489c597ce550dd773d"
I20240501 07:59:15.501061 9188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bc915ffbbf345cbbe1195f8dd3ab874" candidate_uuid: "1f9cb72be98941b5a8e504cbe34132b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83f20e84af4c40d09a4804b6b26e697f"
I20240501 07:59:15.501163 9188 raft_consensus.cc:3037] T 4bc915ffbbf345cbbe1195f8dd3ab874 P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:15.501163 8925 raft_consensus.cc:3037] T 4bc915ffbbf345cbbe1195f8dd3ab874 P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:15.502108 8925 raft_consensus.cc:2454] T 4bc915ffbbf345cbbe1195f8dd3ab874 P 13d05ebdca8c4f489c597ce550dd773d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f9cb72be98941b5a8e504cbe34132b9 in term 1.
I20240501 07:59:15.502115 9188 raft_consensus.cc:2454] T 4bc915ffbbf345cbbe1195f8dd3ab874 P 83f20e84af4c40d09a4804b6b26e697f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f9cb72be98941b5a8e504cbe34132b9 in term 1.
I20240501 07:59:15.502410 6643 leader_election.cc:302] T 4bc915ffbbf345cbbe1195f8dd3ab874 P 1f9cb72be98941b5a8e504cbe34132b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13d05ebdca8c4f489c597ce550dd773d, 1f9cb72be98941b5a8e504cbe34132b9; no voters:
I20240501 07:59:15.502565 11402 raft_consensus.cc:2781] T 4bc915ffbbf345cbbe1195f8dd3ab874 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:15.502745 11402 raft_consensus.cc:686] T 4bc915ffbbf345cbbe1195f8dd3ab874 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 LEADER]: Becoming Leader. State: Replica: 1f9cb72be98941b5a8e504cbe34132b9, State: Running, Role: LEADER
I20240501 07:59:15.502884 11402 consensus_queue.cc:237] T 4bc915ffbbf345cbbe1195f8dd3ab874 P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } }
I20240501 07:59:15.505887 6171 catalog_manager.cc:5484] T 4bc915ffbbf345cbbe1195f8dd3ab874 P 1f9cb72be98941b5a8e504cbe34132b9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132). New cstate: current_term: 1 leader_uuid: "1f9cb72be98941b5a8e504cbe34132b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } health_report { overall_health: UNKNOWN } } }
I20240501 07:59:15.571893 6171 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55950:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 07:59:15.577144 6295 tablet_service.cc:1467] Processing CreateTablet for tablet e3539c15a7bb4a09a967ad6b6b9601b2 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=7ed9e34cd6994a18a4abe8cbb9b73232]), partition=
I20240501 07:59:15.577144 8774 tablet_service.cc:1467] Processing CreateTablet for tablet e3539c15a7bb4a09a967ad6b6b9601b2 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=7ed9e34cd6994a18a4abe8cbb9b73232]), partition=
I20240501 07:59:15.577200 9035 tablet_service.cc:1467] Processing CreateTablet for tablet e3539c15a7bb4a09a967ad6b6b9601b2 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=7ed9e34cd6994a18a4abe8cbb9b73232]), partition=
I20240501 07:59:15.577445 6295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3539c15a7bb4a09a967ad6b6b9601b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:15.577445 8774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3539c15a7bb4a09a967ad6b6b9601b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:15.577462 9035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3539c15a7bb4a09a967ad6b6b9601b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:15.579319 12655 tablet_bootstrap.cc:492] T e3539c15a7bb4a09a967ad6b6b9601b2 P a505e7cb133c4392bfda8a4a20874ae9: Bootstrap starting.
I20240501 07:59:15.579650 12283 tablet_bootstrap.cc:492] T e3539c15a7bb4a09a967ad6b6b9601b2 P 74077b61d6f54342bf9736b4a9f2eade: Bootstrap starting.
I20240501 07:59:15.580077 12770 tablet_bootstrap.cc:492] T e3539c15a7bb4a09a967ad6b6b9601b2 P 8b8b3ad3aa8f4053b2167dd550f2086f: Bootstrap starting.
I20240501 07:59:15.580318 12655 tablet_bootstrap.cc:654] T e3539c15a7bb4a09a967ad6b6b9601b2 P a505e7cb133c4392bfda8a4a20874ae9: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:15.580495 12283 tablet_bootstrap.cc:654] T e3539c15a7bb4a09a967ad6b6b9601b2 P 74077b61d6f54342bf9736b4a9f2eade: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:15.581024 12770 tablet_bootstrap.cc:654] T e3539c15a7bb4a09a967ad6b6b9601b2 P 8b8b3ad3aa8f4053b2167dd550f2086f: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:15.581959 12655 tablet_bootstrap.cc:492] T e3539c15a7bb4a09a967ad6b6b9601b2 P a505e7cb133c4392bfda8a4a20874ae9: No bootstrap required, opened a new log
I20240501 07:59:15.582069 12655 ts_tablet_manager.cc:1367] T e3539c15a7bb4a09a967ad6b6b9601b2 P a505e7cb133c4392bfda8a4a20874ae9: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240501 07:59:15.582348 12283 tablet_bootstrap.cc:492] T e3539c15a7bb4a09a967ad6b6b9601b2 P 74077b61d6f54342bf9736b4a9f2eade: No bootstrap required, opened a new log
I20240501 07:59:15.582420 12283 ts_tablet_manager.cc:1367] T e3539c15a7bb4a09a967ad6b6b9601b2 P 74077b61d6f54342bf9736b4a9f2eade: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240501 07:59:15.582580 12655 raft_consensus.cc:348] T e3539c15a7bb4a09a967ad6b6b9601b2 P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } }
I20240501 07:59:15.582708 12655 raft_consensus.cc:374] T e3539c15a7bb4a09a967ad6b6b9601b2 P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:15.582742 12655 raft_consensus.cc:729] T e3539c15a7bb4a09a967ad6b6b9601b2 P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a505e7cb133c4392bfda8a4a20874ae9, State: Initialized, Role: FOLLOWER
I20240501 07:59:15.582877 12655 consensus_queue.cc:260] T e3539c15a7bb4a09a967ad6b6b9601b2 P a505e7cb133c4392bfda8a4a20874ae9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } }
I20240501 07:59:15.582871 12283 raft_consensus.cc:348] T e3539c15a7bb4a09a967ad6b6b9601b2 P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } }
I20240501 07:59:15.582957 12283 raft_consensus.cc:374] T e3539c15a7bb4a09a967ad6b6b9601b2 P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:15.582989 12283 raft_consensus.cc:729] T e3539c15a7bb4a09a967ad6b6b9601b2 P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74077b61d6f54342bf9736b4a9f2eade, State: Initialized, Role: FOLLOWER
I20240501 07:59:15.583112 12283 consensus_queue.cc:260] T e3539c15a7bb4a09a967ad6b6b9601b2 P 74077b61d6f54342bf9736b4a9f2eade [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } }
I20240501 07:59:15.583178 12655 ts_tablet_manager.cc:1398] T e3539c15a7bb4a09a967ad6b6b9601b2 P a505e7cb133c4392bfda8a4a20874ae9: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:59:15.583385 12283 ts_tablet_manager.cc:1398] T e3539c15a7bb4a09a967ad6b6b9601b2 P 74077b61d6f54342bf9736b4a9f2eade: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:15.584579 12770 tablet_bootstrap.cc:492] T e3539c15a7bb4a09a967ad6b6b9601b2 P 8b8b3ad3aa8f4053b2167dd550f2086f: No bootstrap required, opened a new log
I20240501 07:59:15.584735 12770 ts_tablet_manager.cc:1367] T e3539c15a7bb4a09a967ad6b6b9601b2 P 8b8b3ad3aa8f4053b2167dd550f2086f: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20240501 07:59:15.585264 12770 raft_consensus.cc:348] T e3539c15a7bb4a09a967ad6b6b9601b2 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } }
I20240501 07:59:15.585378 12770 raft_consensus.cc:374] T e3539c15a7bb4a09a967ad6b6b9601b2 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:15.585477 12770 raft_consensus.cc:729] T e3539c15a7bb4a09a967ad6b6b9601b2 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b8b3ad3aa8f4053b2167dd550f2086f, State: Initialized, Role: FOLLOWER
I20240501 07:59:15.585614 12770 consensus_queue.cc:260] T e3539c15a7bb4a09a967ad6b6b9601b2 P 8b8b3ad3aa8f4053b2167dd550f2086f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } }
I20240501 07:59:15.585981 12770 ts_tablet_manager.cc:1398] T e3539c15a7bb4a09a967ad6b6b9601b2 P 8b8b3ad3aa8f4053b2167dd550f2086f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:15.601948 12492 ts_tablet_manager.cc:902] T 12ce6a21d06c4ba889542758be9dc62d P 3b164cbbc00f4e8eb3254ddd349a3ff9: Initiating tablet copy from peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607)
I20240501 07:59:15.602439 12492 tablet_copy_client.cc:308] T 12ce6a21d06c4ba889542758be9dc62d P 3b164cbbc00f4e8eb3254ddd349a3ff9: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.135:42607
I20240501 07:59:15.602922 9076 tablet_copy_service.cc:140] P a505e7cb133c4392bfda8a4a20874ae9: Received BeginTabletCopySession request for tablet 12ce6a21d06c4ba889542758be9dc62d from peer 3b164cbbc00f4e8eb3254ddd349a3ff9 ({username='slave'} at 127.0.99.131:39545)
I20240501 07:59:15.603039 9076 tablet_copy_service.cc:161] P a505e7cb133c4392bfda8a4a20874ae9: Beginning new tablet copy session on tablet 12ce6a21d06c4ba889542758be9dc62d from peer 3b164cbbc00f4e8eb3254ddd349a3ff9 at {username='slave'} at 127.0.99.131:39545: session id = 3b164cbbc00f4e8eb3254ddd349a3ff9-12ce6a21d06c4ba889542758be9dc62d
I20240501 07:59:15.603902 9076 tablet_copy_source_session.cc:205] T 12ce6a21d06c4ba889542758be9dc62d P a505e7cb133c4392bfda8a4a20874ae9: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:15.604765 12492 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12ce6a21d06c4ba889542758be9dc62d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:15.606916 12492 tablet_copy_client.cc:783] T 12ce6a21d06c4ba889542758be9dc62d P 3b164cbbc00f4e8eb3254ddd349a3ff9: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:15.607070 12492 tablet_copy_client.cc:647] T 12ce6a21d06c4ba889542758be9dc62d P 3b164cbbc00f4e8eb3254ddd349a3ff9: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:15.608096 12492 tablet_copy_client.cc:523] T 12ce6a21d06c4ba889542758be9dc62d P 3b164cbbc00f4e8eb3254ddd349a3ff9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:15.609695 12492 tablet_bootstrap.cc:492] T 12ce6a21d06c4ba889542758be9dc62d P 3b164cbbc00f4e8eb3254ddd349a3ff9: Bootstrap starting.
I20240501 07:59:15.612731 12492 tablet_bootstrap.cc:492] T 12ce6a21d06c4ba889542758be9dc62d P 3b164cbbc00f4e8eb3254ddd349a3ff9: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:15.612931 12492 tablet_bootstrap.cc:492] T 12ce6a21d06c4ba889542758be9dc62d P 3b164cbbc00f4e8eb3254ddd349a3ff9: Bootstrap complete.
I20240501 07:59:15.613090 12492 ts_tablet_manager.cc:1367] T 12ce6a21d06c4ba889542758be9dc62d P 3b164cbbc00f4e8eb3254ddd349a3ff9: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.001s
I20240501 07:59:15.613441 12492 raft_consensus.cc:348] T 12ce6a21d06c4ba889542758be9dc62d P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { replace: true } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: true } }
I20240501 07:59:15.613541 12492 raft_consensus.cc:729] T 12ce6a21d06c4ba889542758be9dc62d P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3b164cbbc00f4e8eb3254ddd349a3ff9, State: Initialized, Role: LEARNER
I20240501 07:59:15.613618 12492 consensus_queue.cc:260] T 12ce6a21d06c4ba889542758be9dc62d P 3b164cbbc00f4e8eb3254ddd349a3ff9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { replace: true } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: true } }
I20240501 07:59:15.614109 12492 ts_tablet_manager.cc:1398] T 12ce6a21d06c4ba889542758be9dc62d P 3b164cbbc00f4e8eb3254ddd349a3ff9: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 07:59:15.614754 9076 tablet_copy_service.cc:342] P a505e7cb133c4392bfda8a4a20874ae9: Request end of tablet copy session 3b164cbbc00f4e8eb3254ddd349a3ff9-12ce6a21d06c4ba889542758be9dc62d received from {username='slave'} at 127.0.99.131:39545
I20240501 07:59:15.614842 9076 tablet_copy_service.cc:433] P a505e7cb133c4392bfda8a4a20874ae9: ending tablet copy session 3b164cbbc00f4e8eb3254ddd349a3ff9-12ce6a21d06c4ba889542758be9dc62d on tablet 12ce6a21d06c4ba889542758be9dc62d with peer 3b164cbbc00f4e8eb3254ddd349a3ff9
I20240501 07:59:15.633829 11119 raft_consensus.cc:1050] T 114b6dbafa1c4c44ac42f1d2293e3125 P 3b164cbbc00f4e8eb3254ddd349a3ff9: attempting to promote NON_VOTER 74077b61d6f54342bf9736b4a9f2eade to VOTER
I20240501 07:59:15.634303 11119 consensus_queue.cc:237] T 114b6dbafa1c4c44ac42f1d2293e3125 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } }
I20240501 07:59:15.635689 6444 raft_consensus.cc:1261] T 114b6dbafa1c4c44ac42f1d2293e3125 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Refusing update from remote peer 3b164cbbc00f4e8eb3254ddd349a3ff9: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 07:59:15.636071 10046 consensus_queue.cc:1035] T 114b6dbafa1c4c44ac42f1d2293e3125 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 07:59:15.636432 8793 raft_consensus.cc:1261] T 114b6dbafa1c4c44ac42f1d2293e3125 P 74077b61d6f54342bf9736b4a9f2eade [term 1 LEARNER]: Refusing update from remote peer 3b164cbbc00f4e8eb3254ddd349a3ff9: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 07:59:15.636747 10046 consensus_queue.cc:1035] T 114b6dbafa1c4c44ac42f1d2293e3125 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 07:59:15.637034 9056 raft_consensus.cc:1261] T 114b6dbafa1c4c44ac42f1d2293e3125 P a505e7cb133c4392bfda8a4a20874ae9 [term 1 FOLLOWER]: Refusing update from remote peer 3b164cbbc00f4e8eb3254ddd349a3ff9: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 07:59:15.638641 12119 consensus_queue.cc:1035] T 114b6dbafa1c4c44ac42f1d2293e3125 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 07:59:15.638913 11119 raft_consensus.cc:2932] T 114b6dbafa1c4c44ac42f1d2293e3125 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } } }
I20240501 07:59:15.639699 9056 raft_consensus.cc:2932] T 114b6dbafa1c4c44ac42f1d2293e3125 P a505e7cb133c4392bfda8a4a20874ae9 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } } }
I20240501 07:59:15.640820 11901 raft_consensus.cc:482] T e3539c15a7bb4a09a967ad6b6b9601b2 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:15.640918 11901 raft_consensus.cc:504] T e3539c15a7bb4a09a967ad6b6b9601b2 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } }
I20240501 07:59:15.641038 6444 raft_consensus.cc:2932] T 114b6dbafa1c4c44ac42f1d2293e3125 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } } }
I20240501 07:59:15.641361 11901 leader_election.cc:288] T e3539c15a7bb4a09a967ad6b6b9601b2 P 8b8b3ad3aa8f4053b2167dd550f2086f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133:39859), a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607)
I20240501 07:59:15.641512 8793 raft_consensus.cc:2932] T 114b6dbafa1c4c44ac42f1d2293e3125 P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } } }
I20240501 07:59:15.643453 6165 catalog_manager.cc:5484] T 114b6dbafa1c4c44ac42f1d2293e3125 P 3b164cbbc00f4e8eb3254ddd349a3ff9 reported cstate change: config changed from index 7 to 8, 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 07:59:15.643677 9055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3539c15a7bb4a09a967ad6b6b9601b2" candidate_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a505e7cb133c4392bfda8a4a20874ae9" is_pre_election: true
I20240501 07:59:15.643800 9055 raft_consensus.cc:2454] T e3539c15a7bb4a09a967ad6b6b9601b2 P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b8b3ad3aa8f4053b2167dd550f2086f in term 0.
I20240501 07:59:15.644096 6249 leader_election.cc:302] T e3539c15a7bb4a09a967ad6b6b9601b2 P 8b8b3ad3aa8f4053b2167dd550f2086f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b8b3ad3aa8f4053b2167dd550f2086f, a505e7cb133c4392bfda8a4a20874ae9; no voters:
I20240501 07:59:15.644269 11901 raft_consensus.cc:2781] T e3539c15a7bb4a09a967ad6b6b9601b2 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:15.644322 11901 raft_consensus.cc:482] T e3539c15a7bb4a09a967ad6b6b9601b2 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:15.644359 11901 raft_consensus.cc:3037] T e3539c15a7bb4a09a967ad6b6b9601b2 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:15.645372 11901 raft_consensus.cc:504] T e3539c15a7bb4a09a967ad6b6b9601b2 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } }
I20240501 07:59:15.645727 11901 leader_election.cc:288] T e3539c15a7bb4a09a967ad6b6b9601b2 P 8b8b3ad3aa8f4053b2167dd550f2086f [CANDIDATE]: Term 1 election: Requested vote from peers 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133:39859), a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607)
I20240501 07:59:15.646466 8793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3539c15a7bb4a09a967ad6b6b9601b2" candidate_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74077b61d6f54342bf9736b4a9f2eade" is_pre_election: true
I20240501 07:59:15.646540 8794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3539c15a7bb4a09a967ad6b6b9601b2" candidate_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74077b61d6f54342bf9736b4a9f2eade"
I20240501 07:59:15.646595 8793 raft_consensus.cc:2454] T e3539c15a7bb4a09a967ad6b6b9601b2 P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b8b3ad3aa8f4053b2167dd550f2086f in term 0.
I20240501 07:59:15.646770 9056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3539c15a7bb4a09a967ad6b6b9601b2" candidate_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a505e7cb133c4392bfda8a4a20874ae9"
I20240501 07:59:15.646896 9056 raft_consensus.cc:3037] T e3539c15a7bb4a09a967ad6b6b9601b2 P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:15.647856 9056 raft_consensus.cc:2454] T e3539c15a7bb4a09a967ad6b6b9601b2 P a505e7cb133c4392bfda8a4a20874ae9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b8b3ad3aa8f4053b2167dd550f2086f in term 1.
I20240501 07:59:15.648125 6249 leader_election.cc:302] T e3539c15a7bb4a09a967ad6b6b9601b2 P 8b8b3ad3aa8f4053b2167dd550f2086f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8b8b3ad3aa8f4053b2167dd550f2086f, a505e7cb133c4392bfda8a4a20874ae9; no voters: 74077b61d6f54342bf9736b4a9f2eade
I20240501 07:59:15.649456 11901 raft_consensus.cc:2781] T e3539c15a7bb4a09a967ad6b6b9601b2 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:15.649545 11901 raft_consensus.cc:686] T e3539c15a7bb4a09a967ad6b6b9601b2 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 LEADER]: Becoming Leader. State: Replica: 8b8b3ad3aa8f4053b2167dd550f2086f, State: Running, Role: LEADER
I20240501 07:59:15.649698 11901 consensus_queue.cc:237] T e3539c15a7bb4a09a967ad6b6b9601b2 P 8b8b3ad3aa8f4053b2167dd550f2086f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } }
I20240501 07:59:15.651813 6171 catalog_manager.cc:5484] T e3539c15a7bb4a09a967ad6b6b9601b2 P 8b8b3ad3aa8f4053b2167dd550f2086f reported cstate change: term changed from 0 to 1, leader changed from <none> to 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129). New cstate: current_term: 1 leader_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } health_report { overall_health: UNKNOWN } } }
I20240501 07:59:15.669883 8793 raft_consensus.cc:1203] T 6221deb431d94bfca1c14696e0e0a0c5 P 74077b61d6f54342bf9736b4a9f2eade [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 07:59:15.708508 12608 consensus_queue.cc:1035] T 8f36eefcda3f475d9571e92fdeb52eb4 P a505e7cb133c4392bfda8a4a20874ae9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:15.714243 12488 consensus_queue.cc:1035] T 8f36eefcda3f475d9571e92fdeb52eb4 P a505e7cb133c4392bfda8a4a20874ae9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:15.743322 12503 consensus_queue.cc:1035] T a93ce5841c7f40a9ab23f88efe7b85bb P 83f20e84af4c40d09a4804b6b26e697f [LEADER]: Connected to new peer: Peer: permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:15.746340 12838 consensus_queue.cc:1035] T a93ce5841c7f40a9ab23f88efe7b85bb P 83f20e84af4c40d09a4804b6b26e697f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:15.764454 6171 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55950:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 07:59:15.766036 6575 raft_consensus.cc:1203] T 59e171892fc1414fbcfd6a8c090854f3 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 07:59:15.770387 9056 raft_consensus.cc:1261] T e3539c15a7bb4a09a967ad6b6b9601b2 P a505e7cb133c4392bfda8a4a20874ae9 [term 1 FOLLOWER]: Refusing update from remote peer 8b8b3ad3aa8f4053b2167dd550f2086f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:15.770401 12842 ts_tablet_manager.cc:902] T b7d57b7fd5234f50afbc859bc997f3e1 P 83f20e84af4c40d09a4804b6b26e697f: Initiating tablet copy from peer 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131:34471)
I20240501 07:59:15.770459 8793 raft_consensus.cc:3037] T e3539c15a7bb4a09a967ad6b6b9601b2 P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:15.770929 12842 tablet_copy_client.cc:308] T b7d57b7fd5234f50afbc859bc997f3e1 P 83f20e84af4c40d09a4804b6b26e697f: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:34471
I20240501 07:59:15.771364 6597 tablet_copy_service.cc:140] P 3b164cbbc00f4e8eb3254ddd349a3ff9: Received BeginTabletCopySession request for tablet b7d57b7fd5234f50afbc859bc997f3e1 from peer 83f20e84af4c40d09a4804b6b26e697f ({username='slave'} at 127.0.99.136:47269)
I20240501 07:59:15.771414 11901 consensus_queue.cc:1035] T e3539c15a7bb4a09a967ad6b6b9601b2 P 8b8b3ad3aa8f4053b2167dd550f2086f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:15.771485 6597 tablet_copy_service.cc:161] P 3b164cbbc00f4e8eb3254ddd349a3ff9: Beginning new tablet copy session on tablet b7d57b7fd5234f50afbc859bc997f3e1 from peer 83f20e84af4c40d09a4804b6b26e697f at {username='slave'} at 127.0.99.136:47269: session id = 83f20e84af4c40d09a4804b6b26e697f-b7d57b7fd5234f50afbc859bc997f3e1
I20240501 07:59:15.771708 8793 raft_consensus.cc:1261] T e3539c15a7bb4a09a967ad6b6b9601b2 P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Refusing update from remote peer 8b8b3ad3aa8f4053b2167dd550f2086f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:15.772150 11901 consensus_queue.cc:1035] T e3539c15a7bb4a09a967ad6b6b9601b2 P 8b8b3ad3aa8f4053b2167dd550f2086f [LEADER]: Connected to new peer: Peer: permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:15.772444 6597 tablet_copy_source_session.cc:205] T b7d57b7fd5234f50afbc859bc997f3e1 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:15.774034 6166 catalog_manager.cc:5576] TS a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607) does not have the latest schema for tablet e3539c15a7bb4a09a967ad6b6b9601b2 (table rebalancer_dynamic_table [id=7ed9e34cd6994a18a4abe8cbb9b73232]). Expected version 1 got 0
I20240501 07:59:15.774654 12849 tablet.cc:1722] T e3539c15a7bb4a09a967ad6b6b9601b2 P 8b8b3ad3aa8f4053b2167dd550f2086f: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 07:59:15.774855 12842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7d57b7fd5234f50afbc859bc997f3e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:15.774981 12849 tablet.cc:1620] T e3539c15a7bb4a09a967ad6b6b9601b2 P 8b8b3ad3aa8f4053b2167dd550f2086f: MemRowSet was empty: no flush needed.
I20240501 07:59:15.775360 6171 catalog_manager.cc:5576] TS 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133:39859) does not have the latest schema for tablet e3539c15a7bb4a09a967ad6b6b9601b2 (table rebalancer_dynamic_table [id=7ed9e34cd6994a18a4abe8cbb9b73232]). Expected version 1 got 0
I20240501 07:59:15.777204 12575 tablet.cc:1722] T e3539c15a7bb4a09a967ad6b6b9601b2 P 74077b61d6f54342bf9736b4a9f2eade: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 07:59:15.777535 12575 tablet.cc:1620] T e3539c15a7bb4a09a967ad6b6b9601b2 P 74077b61d6f54342bf9736b4a9f2eade: MemRowSet was empty: no flush needed.
I20240501 07:59:15.777925 12842 tablet_copy_client.cc:783] T b7d57b7fd5234f50afbc859bc997f3e1 P 83f20e84af4c40d09a4804b6b26e697f: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:15.778223 12842 tablet_copy_client.cc:647] T b7d57b7fd5234f50afbc859bc997f3e1 P 83f20e84af4c40d09a4804b6b26e697f: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:15.778792 6164 catalog_manager.cc:5576] TS a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607) does not have the latest schema for tablet e3539c15a7bb4a09a967ad6b6b9601b2 (table rebalancer_dynamic_table [id=7ed9e34cd6994a18a4abe8cbb9b73232]). Expected version 1 got 0
I20240501 07:59:15.779031 12850 tablet.cc:1722] T e3539c15a7bb4a09a967ad6b6b9601b2 P a505e7cb133c4392bfda8a4a20874ae9: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 07:59:15.779317 12850 tablet.cc:1620] T e3539c15a7bb4a09a967ad6b6b9601b2 P a505e7cb133c4392bfda8a4a20874ae9: MemRowSet was empty: no flush needed.
I20240501 07:59:15.780351 12842 tablet_copy_client.cc:523] T b7d57b7fd5234f50afbc859bc997f3e1 P 83f20e84af4c40d09a4804b6b26e697f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:15.781718 12842 tablet_bootstrap.cc:492] T b7d57b7fd5234f50afbc859bc997f3e1 P 83f20e84af4c40d09a4804b6b26e697f: Bootstrap starting.
I20240501 07:59:15.781936 6171 catalog_manager.cc:5576] TS 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129:46875) does not have the latest schema for tablet e3539c15a7bb4a09a967ad6b6b9601b2 (table rebalancer_dynamic_table [id=7ed9e34cd6994a18a4abe8cbb9b73232]). Expected version 1 got 0
I20240501 07:59:15.784847 6164 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 0bc706e85ce74cef97a4827ea4d4aa65: rebalancer_dynamic_table [id=7ed9e34cd6994a18a4abe8cbb9b73232] alter complete (version 1)
I20240501 07:59:15.802958 12842 tablet_bootstrap.cc:492] T b7d57b7fd5234f50afbc859bc997f3e1 P 83f20e84af4c40d09a4804b6b26e697f: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:15.803293 12842 tablet_bootstrap.cc:492] T b7d57b7fd5234f50afbc859bc997f3e1 P 83f20e84af4c40d09a4804b6b26e697f: Bootstrap complete.
I20240501 07:59:15.803453 12842 ts_tablet_manager.cc:1367] T b7d57b7fd5234f50afbc859bc997f3e1 P 83f20e84af4c40d09a4804b6b26e697f: Time spent bootstrapping tablet: real 0.022s user 0.016s sys 0.003s
I20240501 07:59:15.803830 12842 raft_consensus.cc:348] T b7d57b7fd5234f50afbc859bc997f3e1 P 83f20e84af4c40d09a4804b6b26e697f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: true } }
I20240501 07:59:15.803941 12842 raft_consensus.cc:729] T b7d57b7fd5234f50afbc859bc997f3e1 P 83f20e84af4c40d09a4804b6b26e697f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 83f20e84af4c40d09a4804b6b26e697f, State: Initialized, Role: LEARNER
I20240501 07:59:15.804049 12842 consensus_queue.cc:260] T b7d57b7fd5234f50afbc859bc997f3e1 P 83f20e84af4c40d09a4804b6b26e697f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: true } }
I20240501 07:59:15.804337 12842 ts_tablet_manager.cc:1398] T b7d57b7fd5234f50afbc859bc997f3e1 P 83f20e84af4c40d09a4804b6b26e697f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:59:15.804879 6597 tablet_copy_service.cc:342] P 3b164cbbc00f4e8eb3254ddd349a3ff9: Request end of tablet copy session 83f20e84af4c40d09a4804b6b26e697f-b7d57b7fd5234f50afbc859bc997f3e1 received from {username='slave'} at 127.0.99.136:47269
I20240501 07:59:15.805006 6597 tablet_copy_service.cc:433] P 3b164cbbc00f4e8eb3254ddd349a3ff9: ending tablet copy session 83f20e84af4c40d09a4804b6b26e697f-b7d57b7fd5234f50afbc859bc997f3e1 on tablet b7d57b7fd5234f50afbc859bc997f3e1 with peer 83f20e84af4c40d09a4804b6b26e697f
I20240501 07:59:15.811694 9188 raft_consensus.cc:1261] T 6dc929e2a6ae45bd8b1f297583636bb6 P 83f20e84af4c40d09a4804b6b26e697f [term 2 FOLLOWER]: Refusing update from remote peer 74077b61d6f54342bf9736b4a9f2eade: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 07:59:15.812188 11723 consensus_queue.cc:1035] T 6dc929e2a6ae45bd8b1f297583636bb6 P 74077b61d6f54342bf9736b4a9f2eade [LEADER]: Connected to new peer: Peer: permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 07:59:15.848201 6313 raft_consensus.cc:1261] T 6dc929e2a6ae45bd8b1f297583636bb6 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 2 FOLLOWER]: Refusing update from remote peer 74077b61d6f54342bf9736b4a9f2eade: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 07:59:15.848695 6444 raft_consensus.cc:1261] T 6dc929e2a6ae45bd8b1f297583636bb6 P 8315d22c56f7439a9e1fc82f153f018c [term 2 FOLLOWER]: Refusing update from remote peer 74077b61d6f54342bf9736b4a9f2eade: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 07:59:15.848922 11723 consensus_queue.cc:1035] T 6dc929e2a6ae45bd8b1f297583636bb6 P 74077b61d6f54342bf9736b4a9f2eade [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 07:59:15.849129 11724 consensus_queue.cc:1035] T 6dc929e2a6ae45bd8b1f297583636bb6 P 74077b61d6f54342bf9736b4a9f2eade [LEADER]: Connected to new peer: Peer: permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 07:59:15.857147 8793 consensus_queue.cc:237] T 6dc929e2a6ae45bd8b1f297583636bb6 P 74077b61d6f54342bf9736b4a9f2eade [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false } }
I20240501 07:59:15.858180 11453 raft_consensus.cc:1050] T 28eaf22342ff4b5394f958d0b659273c P a505e7cb133c4392bfda8a4a20874ae9: attempting to promote NON_VOTER 3b164cbbc00f4e8eb3254ddd349a3ff9 to VOTER
I20240501 07:59:15.858574 11453 consensus_queue.cc:237] T 28eaf22342ff4b5394f958d0b659273c P a505e7cb133c4392bfda8a4a20874ae9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: false } }
I20240501 07:59:15.860106 6575 raft_consensus.cc:1261] T 28eaf22342ff4b5394f958d0b659273c P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 LEARNER]: Refusing update from remote peer a505e7cb133c4392bfda8a4a20874ae9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 07:59:15.860219 6313 raft_consensus.cc:1261] T 6dc929e2a6ae45bd8b1f297583636bb6 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 2 FOLLOWER]: Refusing update from remote peer 74077b61d6f54342bf9736b4a9f2eade: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 07:59:15.860778 12608 consensus_queue.cc:1035] T 28eaf22342ff4b5394f958d0b659273c P a505e7cb133c4392bfda8a4a20874ae9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 07:59:15.861434 11723 consensus_queue.cc:1035] T 6dc929e2a6ae45bd8b1f297583636bb6 P 74077b61d6f54342bf9736b4a9f2eade [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 07:59:15.862907 8793 raft_consensus.cc:1261] T 28eaf22342ff4b5394f958d0b659273c P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Refusing update from remote peer a505e7cb133c4392bfda8a4a20874ae9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 07:59:15.863076 11724 raft_consensus.cc:2932] T 6dc929e2a6ae45bd8b1f297583636bb6 P 74077b61d6f54342bf9736b4a9f2eade [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false } } }
I20240501 07:59:15.863306 12854 consensus_queue.cc:1035] T 28eaf22342ff4b5394f958d0b659273c P a505e7cb133c4392bfda8a4a20874ae9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 07:59:15.863745 6708 raft_consensus.cc:1261] T 28eaf22342ff4b5394f958d0b659273c P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Refusing update from remote peer a505e7cb133c4392bfda8a4a20874ae9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 07:59:15.864866 6157 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6dc929e2a6ae45bd8b1f297583636bb6 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 07:59:15.865190 12608 consensus_queue.cc:1035] T 28eaf22342ff4b5394f958d0b659273c P a505e7cb133c4392bfda8a4a20874ae9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 07:59:15.865259 6313 raft_consensus.cc:2932] T 6dc929e2a6ae45bd8b1f297583636bb6 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false } } }
I20240501 07:59:15.866391 9188 raft_consensus.cc:1261] T 6dc929e2a6ae45bd8b1f297583636bb6 P 83f20e84af4c40d09a4804b6b26e697f [term 2 FOLLOWER]: Refusing update from remote peer 74077b61d6f54342bf9736b4a9f2eade: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 07:59:15.866798 12123 consensus_queue.cc:1035] T 6dc929e2a6ae45bd8b1f297583636bb6 P 74077b61d6f54342bf9736b4a9f2eade [LEADER]: Connected to new peer: Peer: permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 07:59:15.867002 12608 raft_consensus.cc:2932] T 28eaf22342ff4b5394f958d0b659273c P a505e7cb133c4392bfda8a4a20874ae9 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: false } } }
I20240501 07:59:15.867401 6708 raft_consensus.cc:2932] T 28eaf22342ff4b5394f958d0b659273c P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: false } } }
I20240501 07:59:15.867887 9188 raft_consensus.cc:2932] T 6dc929e2a6ae45bd8b1f297583636bb6 P 83f20e84af4c40d09a4804b6b26e697f [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false } } }
I20240501 07:59:15.869688 6575 raft_consensus.cc:2932] T 28eaf22342ff4b5394f958d0b659273c P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: false } } }
I20240501 07:59:15.871464 6165 catalog_manager.cc:5484] T 6dc929e2a6ae45bd8b1f297583636bb6 P 83f20e84af4c40d09a4804b6b26e697f reported cstate change: config changed from index 8 to 10, VOTER 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130) evicted. New cstate: current_term: 2 leader_uuid: "74077b61d6f54342bf9736b4a9f2eade" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false } } }
I20240501 07:59:15.872058 6171 catalog_manager.cc:5484] T 28eaf22342ff4b5394f958d0b659273c P 1f9cb72be98941b5a8e504cbe34132b9 reported cstate change: config changed from index 2 to 3, 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a505e7cb133c4392bfda8a4a20874ae9" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: false } } }
I20240501 07:59:15.874502 8793 raft_consensus.cc:2932] T 28eaf22342ff4b5394f958d0b659273c P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: false } } }
I20240501 07:59:15.876941 6424 tablet_service.cc:1514] Processing DeleteTablet for tablet 6dc929e2a6ae45bd8b1f297583636bb6 with delete_type TABLET_DATA_TOMBSTONED (TS 8315d22c56f7439a9e1fc82f153f018c not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:45034
I20240501 07:59:15.878348 9056 consensus_queue.cc:237] T 28eaf22342ff4b5394f958d0b659273c P a505e7cb133c4392bfda8a4a20874ae9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: false } }
I20240501 07:59:15.879525 6707 raft_consensus.cc:1261] T 28eaf22342ff4b5394f958d0b659273c P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Refusing update from remote peer a505e7cb133c4392bfda8a4a20874ae9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:59:15.880883 6577 raft_consensus.cc:1261] T 28eaf22342ff4b5394f958d0b659273c P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Refusing update from remote peer a505e7cb133c4392bfda8a4a20874ae9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:59:15.880872 12608 consensus_queue.cc:1035] T 28eaf22342ff4b5394f958d0b659273c P a505e7cb133c4392bfda8a4a20874ae9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:15.881322 12608 consensus_queue.cc:1035] T 28eaf22342ff4b5394f958d0b659273c P a505e7cb133c4392bfda8a4a20874ae9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:15.883097 12488 raft_consensus.cc:2932] T 28eaf22342ff4b5394f958d0b659273c P a505e7cb133c4392bfda8a4a20874ae9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: false } } }
I20240501 07:59:15.883379 12858 tablet_replica.cc:330] T 6dc929e2a6ae45bd8b1f297583636bb6 P 8315d22c56f7439a9e1fc82f153f018c: stopping tablet replica
I20240501 07:59:15.883697 6707 raft_consensus.cc:2932] T 28eaf22342ff4b5394f958d0b659273c P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: false } } }
I20240501 07:59:15.884001 12858 raft_consensus.cc:2229] T 6dc929e2a6ae45bd8b1f297583636bb6 P 8315d22c56f7439a9e1fc82f153f018c [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:15.884140 12858 raft_consensus.cc:2258] T 6dc929e2a6ae45bd8b1f297583636bb6 P 8315d22c56f7439a9e1fc82f153f018c [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:15.884909 12858 ts_tablet_manager.cc:1875] T 6dc929e2a6ae45bd8b1f297583636bb6 P 8315d22c56f7439a9e1fc82f153f018c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:59:15.885658 6158 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 28eaf22342ff4b5394f958d0b659273c with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 07:59:15.885941 6166 catalog_manager.cc:5484] T 28eaf22342ff4b5394f958d0b659273c P a505e7cb133c4392bfda8a4a20874ae9 reported cstate change: config changed from index 3 to 4, VOTER 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133) evicted. New cstate: current_term: 1 leader_uuid: "a505e7cb133c4392bfda8a4a20874ae9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 07:59:15.886931 6165 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55950:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 07:59:15.887509 12858 ts_tablet_manager.cc:1888] T 6dc929e2a6ae45bd8b1f297583636bb6 P 8315d22c56f7439a9e1fc82f153f018c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9
I20240501 07:59:15.887588 12858 log.cc:1196] T 6dc929e2a6ae45bd8b1f297583636bb6 P 8315d22c56f7439a9e1fc82f153f018c: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/6dc929e2a6ae45bd8b1f297583636bb6
I20240501 07:59:15.888183 6159 catalog_manager.cc:4830] TS 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130:32941): tablet 6dc929e2a6ae45bd8b1f297583636bb6 (table rebalance_test_table_9 [id=c4f46595186f4427aa00d6752aa5dc9a]) successfully deleted
I20240501 07:59:15.890806 8774 tablet_service.cc:1514] Processing DeleteTablet for tablet 28eaf22342ff4b5394f958d0b659273c with delete_type TABLET_DATA_TOMBSTONED (TS 74077b61d6f54342bf9736b4a9f2eade not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:43136
I20240501 07:59:15.891152 12768 tablet_replica.cc:330] T 28eaf22342ff4b5394f958d0b659273c P 74077b61d6f54342bf9736b4a9f2eade: stopping tablet replica
I20240501 07:59:15.891296 12768 raft_consensus.cc:2229] T 28eaf22342ff4b5394f958d0b659273c P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:15.891438 12768 raft_consensus.cc:2258] T 28eaf22342ff4b5394f958d0b659273c P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:15.892618 12768 ts_tablet_manager.cc:1875] T 28eaf22342ff4b5394f958d0b659273c P 74077b61d6f54342bf9736b4a9f2eade: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:59:15.893659 12850 tablet.cc:1722] T e3539c15a7bb4a09a967ad6b6b9601b2 P a505e7cb133c4392bfda8a4a20874ae9: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 07:59:15.894004 12850 tablet.cc:1620] T e3539c15a7bb4a09a967ad6b6b9601b2 P a505e7cb133c4392bfda8a4a20874ae9: MemRowSet was empty: no flush needed.
I20240501 07:59:15.894905 12768 ts_tablet_manager.cc:1888] T 28eaf22342ff4b5394f958d0b659273c P 74077b61d6f54342bf9736b4a9f2eade: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 07:59:15.894973 12768 log.cc:1196] T 28eaf22342ff4b5394f958d0b659273c P 74077b61d6f54342bf9736b4a9f2eade: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/28eaf22342ff4b5394f958d0b659273c
I20240501 07:59:15.895720 6577 raft_consensus.cc:2932] T 28eaf22342ff4b5394f958d0b659273c P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: false } } }
I20240501 07:59:15.897333 6157 catalog_manager.cc:4830] TS 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133:39859): tablet 28eaf22342ff4b5394f958d0b659273c (table rebalancer_extra_table_4 [id=e37d9a776ed7492ba6f616e4727f982b]) successfully deleted
I20240501 07:59:15.899801 12849 tablet.cc:1722] T e3539c15a7bb4a09a967ad6b6b9601b2 P 8b8b3ad3aa8f4053b2167dd550f2086f: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 07:59:15.900295 12849 tablet.cc:1620] T e3539c15a7bb4a09a967ad6b6b9601b2 P 8b8b3ad3aa8f4053b2167dd550f2086f: MemRowSet was empty: no flush needed.
I20240501 07:59:15.901100 6164 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 0bc706e85ce74cef97a4827ea4d4aa65: rebalancer_dynamic_table [id=7ed9e34cd6994a18a4abe8cbb9b73232] alter complete (version 2)
I20240501 07:59:15.901736 12575 tablet.cc:1722] T e3539c15a7bb4a09a967ad6b6b9601b2 P 74077b61d6f54342bf9736b4a9f2eade: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 07:59:15.902071 12575 tablet.cc:1620] T e3539c15a7bb4a09a967ad6b6b9601b2 P 74077b61d6f54342bf9736b4a9f2eade: MemRowSet was empty: no flush needed.
I20240501 07:59:15.930856 11107 raft_consensus.cc:1050] T a4727883b2544ed58b356f41dc8017a9 P 3b164cbbc00f4e8eb3254ddd349a3ff9: attempting to promote NON_VOTER 13d05ebdca8c4f489c597ce550dd773d to VOTER
I20240501 07:59:15.931401 11107 consensus_queue.cc:237] T a4727883b2544ed58b356f41dc8017a9 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false } }
I20240501 07:59:15.933614 6707 raft_consensus.cc:1261] T a4727883b2544ed58b356f41dc8017a9 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Refusing update from remote peer 3b164cbbc00f4e8eb3254ddd349a3ff9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:59:15.933614 6445 raft_consensus.cc:1261] T a4727883b2544ed58b356f41dc8017a9 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Refusing update from remote peer 3b164cbbc00f4e8eb3254ddd349a3ff9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:59:15.934082 8925 raft_consensus.cc:1261] T a4727883b2544ed58b356f41dc8017a9 P 13d05ebdca8c4f489c597ce550dd773d [term 1 LEARNER]: Refusing update from remote peer 3b164cbbc00f4e8eb3254ddd349a3ff9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:59:15.934095 11119 consensus_queue.cc:1035] T a4727883b2544ed58b356f41dc8017a9 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:59:15.934396 12119 consensus_queue.cc:1035] T a4727883b2544ed58b356f41dc8017a9 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:59:15.934548 11119 consensus_queue.cc:1035] T a4727883b2544ed58b356f41dc8017a9 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:59:15.937213 10046 raft_consensus.cc:2932] T a4727883b2544ed58b356f41dc8017a9 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false } } }
I20240501 07:59:15.937789 6445 raft_consensus.cc:2932] T a4727883b2544ed58b356f41dc8017a9 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false } } }
I20240501 07:59:15.939942 8925 raft_consensus.cc:2932] T a4727883b2544ed58b356f41dc8017a9 P 13d05ebdca8c4f489c597ce550dd773d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false } } }
I20240501 07:59:15.940222 6171 catalog_manager.cc:5484] T a4727883b2544ed58b356f41dc8017a9 P 3b164cbbc00f4e8eb3254ddd349a3ff9 reported cstate change: config changed from index 4 to 5, 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 07:59:15.941707 6707 raft_consensus.cc:2932] T a4727883b2544ed58b356f41dc8017a9 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false } } }
I20240501 07:59:15.943799 6577 consensus_queue.cc:237] T a4727883b2544ed58b356f41dc8017a9 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false } }
I20240501 07:59:15.945665 8925 raft_consensus.cc:1261] T a4727883b2544ed58b356f41dc8017a9 P 13d05ebdca8c4f489c597ce550dd773d [term 1 FOLLOWER]: Refusing update from remote peer 3b164cbbc00f4e8eb3254ddd349a3ff9: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 07:59:15.945665 6707 raft_consensus.cc:1261] T a4727883b2544ed58b356f41dc8017a9 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Refusing update from remote peer 3b164cbbc00f4e8eb3254ddd349a3ff9: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 07:59:15.946390 11107 consensus_queue.cc:1035] T a4727883b2544ed58b356f41dc8017a9 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 07:59:15.946743 12119 consensus_queue.cc:1035] T a4727883b2544ed58b356f41dc8017a9 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 07:59:15.948302 10046 raft_consensus.cc:2932] T a4727883b2544ed58b356f41dc8017a9 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false } } }
I20240501 07:59:15.948812 8925 raft_consensus.cc:2932] T a4727883b2544ed58b356f41dc8017a9 P 13d05ebdca8c4f489c597ce550dd773d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false } } }
I20240501 07:59:15.949860 6160 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a4727883b2544ed58b356f41dc8017a9 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 07:59:15.950469 6707 raft_consensus.cc:2932] T a4727883b2544ed58b356f41dc8017a9 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false } } }
I20240501 07:59:15.951242 6164 catalog_manager.cc:5484] T a4727883b2544ed58b356f41dc8017a9 P 3b164cbbc00f4e8eb3254ddd349a3ff9 reported cstate change: config changed from index 5 to 6, VOTER 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 07:59:15.952355 12123 raft_consensus.cc:1050] T 2f9e82c3f1df4d1292e37e9cea07186e P 74077b61d6f54342bf9736b4a9f2eade: attempting to promote NON_VOTER 8315d22c56f7439a9e1fc82f153f018c to VOTER
I20240501 07:59:15.952777 12123 consensus_queue.cc:237] T 2f9e82c3f1df4d1292e37e9cea07186e P 74077b61d6f54342bf9736b4a9f2eade [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { replace: true } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { promote: false } }
I20240501 07:59:15.954209 8924 raft_consensus.cc:1261] T 2f9e82c3f1df4d1292e37e9cea07186e P 13d05ebdca8c4f489c597ce550dd773d [term 1 FOLLOWER]: Refusing update from remote peer 74077b61d6f54342bf9736b4a9f2eade: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 07:59:15.954223 6313 raft_consensus.cc:1261] T 2f9e82c3f1df4d1292e37e9cea07186e P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Refusing update from remote peer 74077b61d6f54342bf9736b4a9f2eade: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 07:59:15.954208 11719 consensus_queue.cc:1035] T 4bc915ffbbf345cbbe1195f8dd3ab874 P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:15.954630 12852 consensus_queue.cc:1035] T 2f9e82c3f1df4d1292e37e9cea07186e P 74077b61d6f54342bf9736b4a9f2eade [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 07:59:15.955801 6424 tablet_service.cc:1514] Processing DeleteTablet for tablet a4727883b2544ed58b356f41dc8017a9 with delete_type TABLET_DATA_TOMBSTONED (TS 8315d22c56f7439a9e1fc82f153f018c not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:45034
I20240501 07:59:15.956012 12858 tablet_replica.cc:330] T a4727883b2544ed58b356f41dc8017a9 P 8315d22c56f7439a9e1fc82f153f018c: stopping tablet replica
I20240501 07:59:15.955965 12048 consensus_queue.cc:1035] T 4bc915ffbbf345cbbe1195f8dd3ab874 P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:15.956151 12858 raft_consensus.cc:2229] T a4727883b2544ed58b356f41dc8017a9 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:15.956307 12858 raft_consensus.cc:2258] T a4727883b2544ed58b356f41dc8017a9 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:15.956815 6445 raft_consensus.cc:1261] T 2f9e82c3f1df4d1292e37e9cea07186e P 8315d22c56f7439a9e1fc82f153f018c [term 1 LEARNER]: Refusing update from remote peer 74077b61d6f54342bf9736b4a9f2eade: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 07:59:15.957840 12858 ts_tablet_manager.cc:1875] T a4727883b2544ed58b356f41dc8017a9 P 8315d22c56f7439a9e1fc82f153f018c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:59:15.958020 11724 consensus_queue.cc:1035] T 2f9e82c3f1df4d1292e37e9cea07186e P 74077b61d6f54342bf9736b4a9f2eade [LEADER]: Connected to new peer: Peer: permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 07:59:15.958900 11724 consensus_queue.cc:1035] T 2f9e82c3f1df4d1292e37e9cea07186e P 74077b61d6f54342bf9736b4a9f2eade [LEADER]: Connected to new peer: Peer: permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 07:59:15.960266 12858 ts_tablet_manager.cc:1888] T a4727883b2544ed58b356f41dc8017a9 P 8315d22c56f7439a9e1fc82f153f018c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 07:59:15.960335 12858 log.cc:1196] T a4727883b2544ed58b356f41dc8017a9 P 8315d22c56f7439a9e1fc82f153f018c: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/a4727883b2544ed58b356f41dc8017a9
I20240501 07:59:15.961920 12852 raft_consensus.cc:2932] T 2f9e82c3f1df4d1292e37e9cea07186e P 74077b61d6f54342bf9736b4a9f2eade [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { replace: true } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { promote: false } } }
I20240501 07:59:15.962250 6159 catalog_manager.cc:4830] TS 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130:32941): tablet a4727883b2544ed58b356f41dc8017a9 (table rebalance_test_table_11 [id=3e062fd8d58941efbd3d190c28bb0059]) successfully deleted
I20240501 07:59:15.962375 6445 raft_consensus.cc:2932] T 2f9e82c3f1df4d1292e37e9cea07186e P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { replace: true } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { promote: false } } }
I20240501 07:59:15.964931 6313 raft_consensus.cc:2932] T 2f9e82c3f1df4d1292e37e9cea07186e P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { replace: true } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { promote: false } } }
I20240501 07:59:15.965119 6171 catalog_manager.cc:5484] T 2f9e82c3f1df4d1292e37e9cea07186e P 74077b61d6f54342bf9736b4a9f2eade reported cstate change: config changed from index 2 to 3, 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "74077b61d6f54342bf9736b4a9f2eade" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 07:59:15.968412 8925 raft_consensus.cc:2932] T 2f9e82c3f1df4d1292e37e9cea07186e P 13d05ebdca8c4f489c597ce550dd773d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { replace: true } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { promote: false } } }
I20240501 07:59:15.970429 8793 consensus_queue.cc:237] T 2f9e82c3f1df4d1292e37e9cea07186e P 74077b61d6f54342bf9736b4a9f2eade [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { promote: false } }
I20240501 07:59:15.971936 6445 raft_consensus.cc:1261] T 2f9e82c3f1df4d1292e37e9cea07186e P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Refusing update from remote peer 74077b61d6f54342bf9736b4a9f2eade: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:59:15.972447 12852 consensus_queue.cc:1035] T 2f9e82c3f1df4d1292e37e9cea07186e P 74077b61d6f54342bf9736b4a9f2eade [LEADER]: Connected to new peer: Peer: permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:15.972559 6313 raft_consensus.cc:1261] T 2f9e82c3f1df4d1292e37e9cea07186e P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Refusing update from remote peer 74077b61d6f54342bf9736b4a9f2eade: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:59:15.973142 12852 consensus_queue.cc:1035] T 2f9e82c3f1df4d1292e37e9cea07186e P 74077b61d6f54342bf9736b4a9f2eade [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:15.974330 12852 raft_consensus.cc:2932] T 2f9e82c3f1df4d1292e37e9cea07186e P 74077b61d6f54342bf9736b4a9f2eade [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { promote: false } } }
I20240501 07:59:15.975885 6157 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 2f9e82c3f1df4d1292e37e9cea07186e with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 07:59:15.977829 6164 catalog_manager.cc:5484] T 2f9e82c3f1df4d1292e37e9cea07186e P 74077b61d6f54342bf9736b4a9f2eade reported cstate change: config changed from index 3 to 4, VOTER 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134) evicted. New cstate: current_term: 1 leader_uuid: "74077b61d6f54342bf9736b4a9f2eade" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 07:59:15.977897 6445 raft_consensus.cc:2932] T 2f9e82c3f1df4d1292e37e9cea07186e P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { promote: false } } }
I20240501 07:59:15.977964 6313 raft_consensus.cc:2932] T 2f9e82c3f1df4d1292e37e9cea07186e P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { promote: false } } }
I20240501 07:59:15.980677 8904 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f9e82c3f1df4d1292e37e9cea07186e with delete_type TABLET_DATA_TOMBSTONED (TS 13d05ebdca8c4f489c597ce550dd773d not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:47202
I20240501 07:59:15.982106 12891 tablet_replica.cc:330] T 2f9e82c3f1df4d1292e37e9cea07186e P 13d05ebdca8c4f489c597ce550dd773d: stopping tablet replica
I20240501 07:59:15.982335 12891 raft_consensus.cc:2229] T 2f9e82c3f1df4d1292e37e9cea07186e P 13d05ebdca8c4f489c597ce550dd773d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:15.982545 12891 raft_consensus.cc:2258] T 2f9e82c3f1df4d1292e37e9cea07186e P 13d05ebdca8c4f489c597ce550dd773d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:15.983491 12891 ts_tablet_manager.cc:1875] T 2f9e82c3f1df4d1292e37e9cea07186e P 13d05ebdca8c4f489c597ce550dd773d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:59:15.986007 12891 ts_tablet_manager.cc:1888] T 2f9e82c3f1df4d1292e37e9cea07186e P 13d05ebdca8c4f489c597ce550dd773d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 07:59:15.986081 12891 log.cc:1196] T 2f9e82c3f1df4d1292e37e9cea07186e P 13d05ebdca8c4f489c597ce550dd773d: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/2f9e82c3f1df4d1292e37e9cea07186e
I20240501 07:59:15.986621 6159 catalog_manager.cc:4830] TS 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134:40141): tablet 2f9e82c3f1df4d1292e37e9cea07186e (table rebalancer_extra_table_6 [id=8b10e9a527644d40b1515561fc5df5c2]) successfully deleted
I20240501 07:59:16.004786 6171 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55950:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 07:59:16.016482 12850 tablet.cc:1722] T e3539c15a7bb4a09a967ad6b6b9601b2 P a505e7cb133c4392bfda8a4a20874ae9: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 07:59:16.021248 12849 tablet.cc:1722] T e3539c15a7bb4a09a967ad6b6b9601b2 P 8b8b3ad3aa8f4053b2167dd550f2086f: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 07:59:16.021961 12575 tablet.cc:1722] T e3539c15a7bb4a09a967ad6b6b9601b2 P 74077b61d6f54342bf9736b4a9f2eade: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 07:59:16.024122 6171 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 0bc706e85ce74cef97a4827ea4d4aa65: rebalancer_dynamic_table_new_name [id=7ed9e34cd6994a18a4abe8cbb9b73232] alter complete (version 3)
I20240501 07:59:16.041725 12159 raft_consensus.cc:1050] T 6c3c0d6fe41f4ae196ae29beb9b68125 P 8b8b3ad3aa8f4053b2167dd550f2086f: attempting to promote NON_VOTER 13d05ebdca8c4f489c597ce550dd773d to VOTER
I20240501 07:59:16.042131 6445 consensus_queue.cc:237] T c3a9036ccfee45498bca9681518d3699 P 8315d22c56f7439a9e1fc82f153f018c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: true } }
I20240501 07:59:16.042197 12159 consensus_queue.cc:237] T 6c3c0d6fe41f4ae196ae29beb9b68125 P 8b8b3ad3aa8f4053b2167dd550f2086f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false } }
I20240501 07:59:16.043562 8925 raft_consensus.cc:1261] T 6c3c0d6fe41f4ae196ae29beb9b68125 P 13d05ebdca8c4f489c597ce550dd773d [term 1 LEARNER]: Refusing update from remote peer 8b8b3ad3aa8f4053b2167dd550f2086f: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 07:59:16.043761 9056 raft_consensus.cc:1261] T 6c3c0d6fe41f4ae196ae29beb9b68125 P a505e7cb133c4392bfda8a4a20874ae9 [term 1 FOLLOWER]: Refusing update from remote peer 8b8b3ad3aa8f4053b2167dd550f2086f: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 07:59:16.044003 6577 raft_consensus.cc:1261] T 6c3c0d6fe41f4ae196ae29beb9b68125 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Refusing update from remote peer 8b8b3ad3aa8f4053b2167dd550f2086f: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 07:59:16.043998 12535 consensus_queue.cc:1035] T 6c3c0d6fe41f4ae196ae29beb9b68125 P 8b8b3ad3aa8f4053b2167dd550f2086f [LEADER]: Connected to new peer: Peer: permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 07:59:16.044433 12159 consensus_queue.cc:1035] T 6c3c0d6fe41f4ae196ae29beb9b68125 P 8b8b3ad3aa8f4053b2167dd550f2086f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240501 07:59:16.044521 6380 consensus_peers.cc:485] T c3a9036ccfee45498bca9681518d3699 P 8315d22c56f7439a9e1fc82f153f018c -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c3a9036ccfee45498bca9681518d3699. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:16.044643 12535 consensus_queue.cc:1035] T 6c3c0d6fe41f4ae196ae29beb9b68125 P 8b8b3ad3aa8f4053b2167dd550f2086f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 07:59:16.044819 8793 raft_consensus.cc:1261] T c3a9036ccfee45498bca9681518d3699 P 74077b61d6f54342bf9736b4a9f2eade [term 2 FOLLOWER]: Refusing update from remote peer 8315d22c56f7439a9e1fc82f153f018c: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240501 07:59:16.045218 11735 consensus_queue.cc:1035] T c3a9036ccfee45498bca9681518d3699 P 8315d22c56f7439a9e1fc82f153f018c [LEADER]: Connected to new peer: Peer: permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 07:59:16.045830 6575 raft_consensus.cc:1261] T c3a9036ccfee45498bca9681518d3699 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 2 FOLLOWER]: Refusing update from remote peer 8315d22c56f7439a9e1fc82f153f018c: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240501 07:59:16.046643 11735 consensus_queue.cc:1035] T c3a9036ccfee45498bca9681518d3699 P 8315d22c56f7439a9e1fc82f153f018c [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 07:59:16.047932 12159 raft_consensus.cc:2932] T 6c3c0d6fe41f4ae196ae29beb9b68125 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false } } }
I20240501 07:59:16.048857 9056 raft_consensus.cc:2932] T 6c3c0d6fe41f4ae196ae29beb9b68125 P a505e7cb133c4392bfda8a4a20874ae9 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false } } }
I20240501 07:59:16.049714 11735 raft_consensus.cc:2932] T c3a9036ccfee45498bca9681518d3699 P 8315d22c56f7439a9e1fc82f153f018c [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: true } } }
I20240501 07:59:16.050509 6575 raft_consensus.cc:2932] T 6c3c0d6fe41f4ae196ae29beb9b68125 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false } } }
I20240501 07:59:16.052186 6164 catalog_manager.cc:5484] T 6c3c0d6fe41f4ae196ae29beb9b68125 P 8b8b3ad3aa8f4053b2167dd550f2086f reported cstate change: config changed from index 7 to 8, 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 07:59:16.053380 6171 catalog_manager.cc:5484] T c3a9036ccfee45498bca9681518d3699 P 8315d22c56f7439a9e1fc82f153f018c reported cstate change: config changed from index 7 to 8, NON_VOTER 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136) added. New cstate: current_term: 2 leader_uuid: "8315d22c56f7439a9e1fc82f153f018c" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 07:59:16.055426 6575 raft_consensus.cc:2932] T c3a9036ccfee45498bca9681518d3699 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: true } } }
I20240501 07:59:16.057497 8925 raft_consensus.cc:2932] T 6c3c0d6fe41f4ae196ae29beb9b68125 P 13d05ebdca8c4f489c597ce550dd773d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false } } }
I20240501 07:59:16.059386 8793 raft_consensus.cc:2932] T c3a9036ccfee45498bca9681518d3699 P 74077b61d6f54342bf9736b4a9f2eade [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: true } } }
I20240501 07:59:16.066329 6170 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55950:
name: "rebalancer_extra_table_22"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 07:59:16.067894 6313 consensus_queue.cc:237] T 6c3c0d6fe41f4ae196ae29beb9b68125 P 8b8b3ad3aa8f4053b2167dd550f2086f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false } }
I20240501 07:59:16.069622 8925 raft_consensus.cc:1261] T 6c3c0d6fe41f4ae196ae29beb9b68125 P 13d05ebdca8c4f489c597ce550dd773d [term 1 FOLLOWER]: Refusing update from remote peer 8b8b3ad3aa8f4053b2167dd550f2086f: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 07:59:16.069777 9056 raft_consensus.cc:1261] T 6c3c0d6fe41f4ae196ae29beb9b68125 P a505e7cb133c4392bfda8a4a20874ae9 [term 1 FOLLOWER]: Refusing update from remote peer 8b8b3ad3aa8f4053b2167dd550f2086f: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 07:59:16.070035 11901 consensus_queue.cc:1035] T 6c3c0d6fe41f4ae196ae29beb9b68125 P 8b8b3ad3aa8f4053b2167dd550f2086f [LEADER]: Connected to new peer: Peer: permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 07:59:16.070317 12535 consensus_queue.cc:1035] T 6c3c0d6fe41f4ae196ae29beb9b68125 P 8b8b3ad3aa8f4053b2167dd550f2086f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 07:59:16.072003 11901 raft_consensus.cc:2932] T 6c3c0d6fe41f4ae196ae29beb9b68125 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false } } }
I20240501 07:59:16.072332 8925 raft_consensus.cc:2932] T 6c3c0d6fe41f4ae196ae29beb9b68125 P 13d05ebdca8c4f489c597ce550dd773d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false } } }
I20240501 07:59:16.073529 9056 raft_consensus.cc:2932] T 6c3c0d6fe41f4ae196ae29beb9b68125 P a505e7cb133c4392bfda8a4a20874ae9 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false } } }
I20240501 07:59:16.073691 6157 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6c3c0d6fe41f4ae196ae29beb9b68125 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 07:59:16.075340 6165 catalog_manager.cc:5484] T 6c3c0d6fe41f4ae196ae29beb9b68125 P 8b8b3ad3aa8f4053b2167dd550f2086f reported cstate change: config changed from index 8 to 9, VOTER 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 07:59:16.079177 6556 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c3c0d6fe41f4ae196ae29beb9b68125 with delete_type TABLET_DATA_TOMBSTONED (TS 3b164cbbc00f4e8eb3254ddd349a3ff9 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:53874
I20240501 07:59:16.079957 12948 tablet_replica.cc:330] T 6c3c0d6fe41f4ae196ae29beb9b68125 P 3b164cbbc00f4e8eb3254ddd349a3ff9: stopping tablet replica
I20240501 07:59:16.080917 8774 tablet_service.cc:1467] Processing CreateTablet for tablet 0b24f67cf1a64debb23ee6618c820d33 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=5908f6a818b8456085a9c05480385528]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:16.080989 6556 tablet_service.cc:1467] Processing CreateTablet for tablet 0b24f67cf1a64debb23ee6618c820d33 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=5908f6a818b8456085a9c05480385528]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:16.081247 8774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b24f67cf1a64debb23ee6618c820d33. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:16.081259 6556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b24f67cf1a64debb23ee6618c820d33. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:16.081473 8757 tablet_service.cc:1467] Processing CreateTablet for tablet c16b7962c18d4e02b850c7c7c678eae4 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=5908f6a818b8456085a9c05480385528]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:16.081697 8757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c16b7962c18d4e02b850c7c7c678eae4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:16.083096 12283 tablet_bootstrap.cc:492] T 0b24f67cf1a64debb23ee6618c820d33 P 74077b61d6f54342bf9736b4a9f2eade: Bootstrap starting.
I20240501 07:59:16.083963 6424 tablet_service.cc:1467] Processing CreateTablet for tablet 62936b541ca541faa99eb36bf24c6404 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=5908f6a818b8456085a9c05480385528]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:16.083972 9035 tablet_service.cc:1467] Processing CreateTablet for tablet c16b7962c18d4e02b850c7c7c678eae4 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=5908f6a818b8456085a9c05480385528]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:16.084223 9035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c16b7962c18d4e02b850c7c7c678eae4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:16.084223 6424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62936b541ca541faa99eb36bf24c6404. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:16.084214 8905 tablet_service.cc:1467] Processing CreateTablet for tablet c16b7962c18d4e02b850c7c7c678eae4 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=5908f6a818b8456085a9c05480385528]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:16.084354 12283 tablet_bootstrap.cc:654] T 0b24f67cf1a64debb23ee6618c820d33 P 74077b61d6f54342bf9736b4a9f2eade: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:16.084494 8905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c16b7962c18d4e02b850c7c7c678eae4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:16.085239 6425 tablet_service.cc:1467] Processing CreateTablet for tablet 0b24f67cf1a64debb23ee6618c820d33 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=5908f6a818b8456085a9c05480385528]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:16.085489 6425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b24f67cf1a64debb23ee6618c820d33. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:16.086632 8904 tablet_service.cc:1467] Processing CreateTablet for tablet 62936b541ca541faa99eb36bf24c6404 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=5908f6a818b8456085a9c05480385528]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:16.086711 12951 tablet_bootstrap.cc:492] T c16b7962c18d4e02b850c7c7c678eae4 P a505e7cb133c4392bfda8a4a20874ae9: Bootstrap starting.
I20240501 07:59:16.086755 6687 tablet_service.cc:1467] Processing CreateTablet for tablet 62936b541ca541faa99eb36bf24c6404 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=5908f6a818b8456085a9c05480385528]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:16.086867 8904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62936b541ca541faa99eb36bf24c6404. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:16.087028 6687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62936b541ca541faa99eb36bf24c6404. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:16.088021 12951 tablet_bootstrap.cc:654] T c16b7962c18d4e02b850c7c7c678eae4 P a505e7cb133c4392bfda8a4a20874ae9: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:16.089138 12951 tablet_bootstrap.cc:492] T c16b7962c18d4e02b850c7c7c678eae4 P a505e7cb133c4392bfda8a4a20874ae9: No bootstrap required, opened a new log
I20240501 07:59:16.089221 12951 ts_tablet_manager.cc:1367] T c16b7962c18d4e02b850c7c7c678eae4 P a505e7cb133c4392bfda8a4a20874ae9: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 07:59:16.089689 12951 raft_consensus.cc:348] T c16b7962c18d4e02b850c7c7c678eae4 P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } }
I20240501 07:59:16.089809 12951 raft_consensus.cc:374] T c16b7962c18d4e02b850c7c7c678eae4 P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:16.089841 12951 raft_consensus.cc:729] T c16b7962c18d4e02b850c7c7c678eae4 P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a505e7cb133c4392bfda8a4a20874ae9, State: Initialized, Role: FOLLOWER
I20240501 07:59:16.089969 12951 consensus_queue.cc:260] T c16b7962c18d4e02b850c7c7c678eae4 P a505e7cb133c4392bfda8a4a20874ae9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } }
I20240501 07:59:16.090247 12951 ts_tablet_manager.cc:1398] T c16b7962c18d4e02b850c7c7c678eae4 P a505e7cb133c4392bfda8a4a20874ae9: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 07:59:16.090708 12948 raft_consensus.cc:2229] T 6c3c0d6fe41f4ae196ae29beb9b68125 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:16.091881 12948 raft_consensus.cc:2258] T 6c3c0d6fe41f4ae196ae29beb9b68125 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:16.093775 12948 ts_tablet_manager.cc:1875] T 6c3c0d6fe41f4ae196ae29beb9b68125 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:59:16.093952 12953 tablet_bootstrap.cc:492] T 0b24f67cf1a64debb23ee6618c820d33 P 8315d22c56f7439a9e1fc82f153f018c: Bootstrap starting.
I20240501 07:59:16.095604 12953 tablet_bootstrap.cc:654] T 0b24f67cf1a64debb23ee6618c820d33 P 8315d22c56f7439a9e1fc82f153f018c: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:16.096205 12952 tablet_bootstrap.cc:492] T c16b7962c18d4e02b850c7c7c678eae4 P 13d05ebdca8c4f489c597ce550dd773d: Bootstrap starting.
I20240501 07:59:16.096467 12949 tablet_bootstrap.cc:492] T 0b24f67cf1a64debb23ee6618c820d33 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Bootstrap starting.
I20240501 07:59:16.097247 12952 tablet_bootstrap.cc:654] T c16b7962c18d4e02b850c7c7c678eae4 P 13d05ebdca8c4f489c597ce550dd773d: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:16.097918 12955 tablet_bootstrap.cc:492] T 62936b541ca541faa99eb36bf24c6404 P 1f9cb72be98941b5a8e504cbe34132b9: Bootstrap starting.
I20240501 07:59:16.098563 12949 tablet_bootstrap.cc:654] T 0b24f67cf1a64debb23ee6618c820d33 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:16.099480 12955 tablet_bootstrap.cc:654] T 62936b541ca541faa99eb36bf24c6404 P 1f9cb72be98941b5a8e504cbe34132b9: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:16.098569 12948 ts_tablet_manager.cc:1888] T 6c3c0d6fe41f4ae196ae29beb9b68125 P 3b164cbbc00f4e8eb3254ddd349a3ff9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240501 07:59:16.100306 12948 log.cc:1196] T 6c3c0d6fe41f4ae196ae29beb9b68125 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/6c3c0d6fe41f4ae196ae29beb9b68125
I20240501 07:59:16.101053 6160 catalog_manager.cc:4830] TS 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131:34471): tablet 6c3c0d6fe41f4ae196ae29beb9b68125 (table rebalance_test_table_19 [id=35f40c7e6b1e4d799576a7d98130d848]) successfully deleted
I20240501 07:59:16.101313 12283 tablet_bootstrap.cc:492] T 0b24f67cf1a64debb23ee6618c820d33 P 74077b61d6f54342bf9736b4a9f2eade: No bootstrap required, opened a new log
I20240501 07:59:16.101393 12283 ts_tablet_manager.cc:1367] T 0b24f67cf1a64debb23ee6618c820d33 P 74077b61d6f54342bf9736b4a9f2eade: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.001s
I20240501 07:59:16.101864 12283 raft_consensus.cc:348] T 0b24f67cf1a64debb23ee6618c820d33 P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } }
I20240501 07:59:16.101997 12283 raft_consensus.cc:374] T 0b24f67cf1a64debb23ee6618c820d33 P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:16.102030 12283 raft_consensus.cc:729] T 0b24f67cf1a64debb23ee6618c820d33 P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74077b61d6f54342bf9736b4a9f2eade, State: Initialized, Role: FOLLOWER
I20240501 07:59:16.102155 12283 consensus_queue.cc:260] T 0b24f67cf1a64debb23ee6618c820d33 P 74077b61d6f54342bf9736b4a9f2eade [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } }
I20240501 07:59:16.102463 12283 ts_tablet_manager.cc:1398] T 0b24f67cf1a64debb23ee6618c820d33 P 74077b61d6f54342bf9736b4a9f2eade: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:16.102892 12283 tablet_bootstrap.cc:492] T c16b7962c18d4e02b850c7c7c678eae4 P 74077b61d6f54342bf9736b4a9f2eade: Bootstrap starting.
I20240501 07:59:16.104470 12283 tablet_bootstrap.cc:654] T c16b7962c18d4e02b850c7c7c678eae4 P 74077b61d6f54342bf9736b4a9f2eade: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:16.105564 12949 tablet_bootstrap.cc:492] T 0b24f67cf1a64debb23ee6618c820d33 P 3b164cbbc00f4e8eb3254ddd349a3ff9: No bootstrap required, opened a new log
I20240501 07:59:16.105741 12949 ts_tablet_manager.cc:1367] T 0b24f67cf1a64debb23ee6618c820d33 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240501 07:59:16.106071 12952 tablet_bootstrap.cc:492] T c16b7962c18d4e02b850c7c7c678eae4 P 13d05ebdca8c4f489c597ce550dd773d: No bootstrap required, opened a new log
I20240501 07:59:16.106168 12952 ts_tablet_manager.cc:1367] T c16b7962c18d4e02b850c7c7c678eae4 P 13d05ebdca8c4f489c597ce550dd773d: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.000s
I20240501 07:59:16.106235 12949 raft_consensus.cc:348] T 0b24f67cf1a64debb23ee6618c820d33 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } }
I20240501 07:59:16.106374 12949 raft_consensus.cc:374] T 0b24f67cf1a64debb23ee6618c820d33 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:16.106475 12949 raft_consensus.cc:729] T 0b24f67cf1a64debb23ee6618c820d33 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b164cbbc00f4e8eb3254ddd349a3ff9, State: Initialized, Role: FOLLOWER
I20240501 07:59:16.106628 12949 consensus_queue.cc:260] T 0b24f67cf1a64debb23ee6618c820d33 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } }
I20240501 07:59:16.106644 12952 raft_consensus.cc:348] T c16b7962c18d4e02b850c7c7c678eae4 P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } }
I20240501 07:59:16.106732 12952 raft_consensus.cc:374] T c16b7962c18d4e02b850c7c7c678eae4 P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:16.106796 12952 raft_consensus.cc:729] T c16b7962c18d4e02b850c7c7c678eae4 P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13d05ebdca8c4f489c597ce550dd773d, State: Initialized, Role: FOLLOWER
I20240501 07:59:16.106923 12952 consensus_queue.cc:260] T c16b7962c18d4e02b850c7c7c678eae4 P 13d05ebdca8c4f489c597ce550dd773d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } }
I20240501 07:59:16.107025 12949 ts_tablet_manager.cc:1398] T 0b24f67cf1a64debb23ee6618c820d33 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:16.107182 12952 ts_tablet_manager.cc:1398] T c16b7962c18d4e02b850c7c7c678eae4 P 13d05ebdca8c4f489c597ce550dd773d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:16.107337 12952 tablet_bootstrap.cc:492] T 62936b541ca541faa99eb36bf24c6404 P 13d05ebdca8c4f489c597ce550dd773d: Bootstrap starting.
I20240501 07:59:16.108172 12952 tablet_bootstrap.cc:654] T 62936b541ca541faa99eb36bf24c6404 P 13d05ebdca8c4f489c597ce550dd773d: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:16.108839 12283 tablet_bootstrap.cc:492] T c16b7962c18d4e02b850c7c7c678eae4 P 74077b61d6f54342bf9736b4a9f2eade: No bootstrap required, opened a new log
I20240501 07:59:16.108939 12283 ts_tablet_manager.cc:1367] T c16b7962c18d4e02b850c7c7c678eae4 P 74077b61d6f54342bf9736b4a9f2eade: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.002s
I20240501 07:59:16.109375 12283 raft_consensus.cc:348] T c16b7962c18d4e02b850c7c7c678eae4 P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } }
I20240501 07:59:16.109476 12283 raft_consensus.cc:374] T c16b7962c18d4e02b850c7c7c678eae4 P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:16.109506 12283 raft_consensus.cc:729] T c16b7962c18d4e02b850c7c7c678eae4 P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74077b61d6f54342bf9736b4a9f2eade, State: Initialized, Role: FOLLOWER
I20240501 07:59:16.109614 12283 consensus_queue.cc:260] T c16b7962c18d4e02b850c7c7c678eae4 P 74077b61d6f54342bf9736b4a9f2eade [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } }
I20240501 07:59:16.109908 12283 ts_tablet_manager.cc:1398] T c16b7962c18d4e02b850c7c7c678eae4 P 74077b61d6f54342bf9736b4a9f2eade: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:16.113626 12955 tablet_bootstrap.cc:492] T 62936b541ca541faa99eb36bf24c6404 P 1f9cb72be98941b5a8e504cbe34132b9: No bootstrap required, opened a new log
I20240501 07:59:16.113690 12953 tablet_bootstrap.cc:492] T 0b24f67cf1a64debb23ee6618c820d33 P 8315d22c56f7439a9e1fc82f153f018c: No bootstrap required, opened a new log
I20240501 07:59:16.113798 12955 ts_tablet_manager.cc:1367] T 62936b541ca541faa99eb36bf24c6404 P 1f9cb72be98941b5a8e504cbe34132b9: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.000s
I20240501 07:59:16.113811 12953 ts_tablet_manager.cc:1367] T 0b24f67cf1a64debb23ee6618c820d33 P 8315d22c56f7439a9e1fc82f153f018c: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.000s
I20240501 07:59:16.114331 12953 raft_consensus.cc:348] T 0b24f67cf1a64debb23ee6618c820d33 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } }
I20240501 07:59:16.114339 12955 raft_consensus.cc:348] T 62936b541ca541faa99eb36bf24c6404 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:59:16.114440 12953 raft_consensus.cc:374] T 0b24f67cf1a64debb23ee6618c820d33 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:16.114454 12955 raft_consensus.cc:374] T 62936b541ca541faa99eb36bf24c6404 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:16.114471 12953 raft_consensus.cc:729] T 0b24f67cf1a64debb23ee6618c820d33 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8315d22c56f7439a9e1fc82f153f018c, State: Initialized, Role: FOLLOWER
I20240501 07:59:16.114480 12955 raft_consensus.cc:729] T 62936b541ca541faa99eb36bf24c6404 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f9cb72be98941b5a8e504cbe34132b9, State: Initialized, Role: FOLLOWER
I20240501 07:59:16.114583 12955 consensus_queue.cc:260] T 62936b541ca541faa99eb36bf24c6404 P 1f9cb72be98941b5a8e504cbe34132b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:59:16.114596 12953 consensus_queue.cc:260] T 0b24f67cf1a64debb23ee6618c820d33 P 8315d22c56f7439a9e1fc82f153f018c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } }
I20240501 07:59:16.114913 12953 ts_tablet_manager.cc:1398] T 0b24f67cf1a64debb23ee6618c820d33 P 8315d22c56f7439a9e1fc82f153f018c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:59:16.115096 12953 tablet_bootstrap.cc:492] T 62936b541ca541faa99eb36bf24c6404 P 8315d22c56f7439a9e1fc82f153f018c: Bootstrap starting.
I20240501 07:59:16.115334 12955 ts_tablet_manager.cc:1398] T 62936b541ca541faa99eb36bf24c6404 P 1f9cb72be98941b5a8e504cbe34132b9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:16.116060 12953 tablet_bootstrap.cc:654] T 62936b541ca541faa99eb36bf24c6404 P 8315d22c56f7439a9e1fc82f153f018c: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:16.116106 12952 tablet_bootstrap.cc:492] T 62936b541ca541faa99eb36bf24c6404 P 13d05ebdca8c4f489c597ce550dd773d: No bootstrap required, opened a new log
I20240501 07:59:16.116192 12952 ts_tablet_manager.cc:1367] T 62936b541ca541faa99eb36bf24c6404 P 13d05ebdca8c4f489c597ce550dd773d: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240501 07:59:16.116789 12952 raft_consensus.cc:348] T 62936b541ca541faa99eb36bf24c6404 P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:59:16.117010 12952 raft_consensus.cc:374] T 62936b541ca541faa99eb36bf24c6404 P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:16.117120 12952 raft_consensus.cc:729] T 62936b541ca541faa99eb36bf24c6404 P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13d05ebdca8c4f489c597ce550dd773d, State: Initialized, Role: FOLLOWER
I20240501 07:59:16.117316 12952 consensus_queue.cc:260] T 62936b541ca541faa99eb36bf24c6404 P 13d05ebdca8c4f489c597ce550dd773d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:59:16.119153 12952 ts_tablet_manager.cc:1398] T 62936b541ca541faa99eb36bf24c6404 P 13d05ebdca8c4f489c597ce550dd773d: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 07:59:16.122354 12953 tablet_bootstrap.cc:492] T 62936b541ca541faa99eb36bf24c6404 P 8315d22c56f7439a9e1fc82f153f018c: No bootstrap required, opened a new log
I20240501 07:59:16.122462 12953 ts_tablet_manager.cc:1367] T 62936b541ca541faa99eb36bf24c6404 P 8315d22c56f7439a9e1fc82f153f018c: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240501 07:59:16.122910 12953 raft_consensus.cc:348] T 62936b541ca541faa99eb36bf24c6404 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:59:16.123020 12953 raft_consensus.cc:374] T 62936b541ca541faa99eb36bf24c6404 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:16.123051 12953 raft_consensus.cc:729] T 62936b541ca541faa99eb36bf24c6404 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8315d22c56f7439a9e1fc82f153f018c, State: Initialized, Role: FOLLOWER
I20240501 07:59:16.123174 12953 consensus_queue.cc:260] T 62936b541ca541faa99eb36bf24c6404 P 8315d22c56f7439a9e1fc82f153f018c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:59:16.123430 12953 ts_tablet_manager.cc:1398] T 62936b541ca541faa99eb36bf24c6404 P 8315d22c56f7439a9e1fc82f153f018c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:16.127471 6166 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55950:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 07:59:16.127616 6166 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55950:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 07:59:16.129843 6166 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 0bc706e85ce74cef97a4827ea4d4aa65: Sending DeleteTablet for 3 replicas of tablet e3539c15a7bb4a09a967ad6b6b9601b2
I20240501 07:59:16.130422 6295 tablet_service.cc:1514] Processing DeleteTablet for tablet e3539c15a7bb4a09a967ad6b6b9601b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:16 UTC) from {username='slave'} at 127.0.0.1:45480
I20240501 07:59:16.130561 8757 tablet_service.cc:1514] Processing DeleteTablet for tablet e3539c15a7bb4a09a967ad6b6b9601b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:16 UTC) from {username='slave'} at 127.0.0.1:43136
I20240501 07:59:16.130578 9035 tablet_service.cc:1514] Processing DeleteTablet for tablet e3539c15a7bb4a09a967ad6b6b9601b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:16 UTC) from {username='slave'} at 127.0.0.1:44236
I20240501 07:59:16.130892 12979 tablet_replica.cc:330] T e3539c15a7bb4a09a967ad6b6b9601b2 P a505e7cb133c4392bfda8a4a20874ae9: stopping tablet replica
I20240501 07:59:16.131075 12979 raft_consensus.cc:2229] T e3539c15a7bb4a09a967ad6b6b9601b2 P a505e7cb133c4392bfda8a4a20874ae9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:16.131198 12979 raft_consensus.cc:2258] T e3539c15a7bb4a09a967ad6b6b9601b2 P a505e7cb133c4392bfda8a4a20874ae9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:16.131518 12768 tablet_replica.cc:330] T e3539c15a7bb4a09a967ad6b6b9601b2 P 74077b61d6f54342bf9736b4a9f2eade: stopping tablet replica
I20240501 07:59:16.131673 12768 raft_consensus.cc:2229] T e3539c15a7bb4a09a967ad6b6b9601b2 P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:16.131713 12979 ts_tablet_manager.cc:1875] T e3539c15a7bb4a09a967ad6b6b9601b2 P a505e7cb133c4392bfda8a4a20874ae9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:59:16.131799 12768 raft_consensus.cc:2258] T e3539c15a7bb4a09a967ad6b6b9601b2 P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:16.132468 12768 ts_tablet_manager.cc:1875] T e3539c15a7bb4a09a967ad6b6b9601b2 P 74077b61d6f54342bf9736b4a9f2eade: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:59:16.134752 12768 ts_tablet_manager.cc:1888] T e3539c15a7bb4a09a967ad6b6b9601b2 P 74077b61d6f54342bf9736b4a9f2eade: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:59:16.134827 12768 log.cc:1196] T e3539c15a7bb4a09a967ad6b6b9601b2 P 74077b61d6f54342bf9736b4a9f2eade: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/e3539c15a7bb4a09a967ad6b6b9601b2
I20240501 07:59:16.135157 12768 ts_tablet_manager.cc:1909] T e3539c15a7bb4a09a967ad6b6b9601b2 P 74077b61d6f54342bf9736b4a9f2eade: Deleting consensus metadata
I20240501 07:59:16.135479 12979 ts_tablet_manager.cc:1888] T e3539c15a7bb4a09a967ad6b6b9601b2 P a505e7cb133c4392bfda8a4a20874ae9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:59:16.135536 12979 log.cc:1196] T e3539c15a7bb4a09a967ad6b6b9601b2 P a505e7cb133c4392bfda8a4a20874ae9: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/e3539c15a7bb4a09a967ad6b6b9601b2
I20240501 07:59:16.135773 12979 ts_tablet_manager.cc:1909] T e3539c15a7bb4a09a967ad6b6b9601b2 P a505e7cb133c4392bfda8a4a20874ae9: Deleting consensus metadata
I20240501 07:59:16.135847 6157 catalog_manager.cc:4830] TS 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133:39859): tablet e3539c15a7bb4a09a967ad6b6b9601b2 (table rebalancer_dynamic_table_new_name [id=7ed9e34cd6994a18a4abe8cbb9b73232]) successfully deleted
I20240501 07:59:16.136212 12978 tablet_replica.cc:330] T e3539c15a7bb4a09a967ad6b6b9601b2 P 8b8b3ad3aa8f4053b2167dd550f2086f: stopping tablet replica
I20240501 07:59:16.136366 12978 raft_consensus.cc:2229] T e3539c15a7bb4a09a967ad6b6b9601b2 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 LEADER]: Raft consensus shutting down.
I20240501 07:59:16.136503 11107 raft_consensus.cc:482] T 0b24f67cf1a64debb23ee6618c820d33 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:16.136628 11107 raft_consensus.cc:504] T 0b24f67cf1a64debb23ee6618c820d33 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } }
I20240501 07:59:16.136754 12978 raft_consensus.cc:2258] T e3539c15a7bb4a09a967ad6b6b9601b2 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:16.137113 11107 leader_election.cc:288] T 0b24f67cf1a64debb23ee6618c820d33 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133:39859), 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130:32941)
I20240501 07:59:16.137476 12978 ts_tablet_manager.cc:1875] T e3539c15a7bb4a09a967ad6b6b9601b2 P 8b8b3ad3aa8f4053b2167dd550f2086f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:59:16.137516 8793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b24f67cf1a64debb23ee6618c820d33" candidate_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74077b61d6f54342bf9736b4a9f2eade" is_pre_election: true
I20240501 07:59:16.137665 8793 raft_consensus.cc:2454] T 0b24f67cf1a64debb23ee6618c820d33 P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b164cbbc00f4e8eb3254ddd349a3ff9 in term 0.
I20240501 07:59:16.137667 6445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b24f67cf1a64debb23ee6618c820d33" candidate_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8315d22c56f7439a9e1fc82f153f018c" is_pre_election: true
I20240501 07:59:16.137816 6445 raft_consensus.cc:2454] T 0b24f67cf1a64debb23ee6618c820d33 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b164cbbc00f4e8eb3254ddd349a3ff9 in term 0.
I20240501 07:59:16.137851 6158 catalog_manager.cc:4830] TS a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): tablet e3539c15a7bb4a09a967ad6b6b9601b2 (table rebalancer_dynamic_table_new_name [id=7ed9e34cd6994a18a4abe8cbb9b73232]) successfully deleted
I20240501 07:59:16.138118 6512 leader_election.cc:302] T 0b24f67cf1a64debb23ee6618c820d33 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b164cbbc00f4e8eb3254ddd349a3ff9, 8315d22c56f7439a9e1fc82f153f018c; no voters:
I20240501 07:59:16.138305 11107 raft_consensus.cc:2781] T 0b24f67cf1a64debb23ee6618c820d33 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:16.138370 11107 raft_consensus.cc:482] T 0b24f67cf1a64debb23ee6618c820d33 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:16.138417 11107 raft_consensus.cc:3037] T 0b24f67cf1a64debb23ee6618c820d33 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:16.139346 11107 raft_consensus.cc:504] T 0b24f67cf1a64debb23ee6618c820d33 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } }
I20240501 07:59:16.139734 12978 ts_tablet_manager.cc:1888] T e3539c15a7bb4a09a967ad6b6b9601b2 P 8b8b3ad3aa8f4053b2167dd550f2086f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:59:16.139746 11107 leader_election.cc:288] T 0b24f67cf1a64debb23ee6618c820d33 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [CANDIDATE]: Term 1 election: Requested vote from peers 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133:39859), 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130:32941)
I20240501 07:59:16.139797 12978 log.cc:1196] T e3539c15a7bb4a09a967ad6b6b9601b2 P 8b8b3ad3aa8f4053b2167dd550f2086f: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/e3539c15a7bb4a09a967ad6b6b9601b2
I20240501 07:59:16.140046 8793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b24f67cf1a64debb23ee6618c820d33" candidate_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74077b61d6f54342bf9736b4a9f2eade"
I20240501 07:59:16.140127 12978 ts_tablet_manager.cc:1909] T e3539c15a7bb4a09a967ad6b6b9601b2 P 8b8b3ad3aa8f4053b2167dd550f2086f: Deleting consensus metadata
I20240501 07:59:16.140168 8793 raft_consensus.cc:3037] T 0b24f67cf1a64debb23ee6618c820d33 P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:16.140151 6445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b24f67cf1a64debb23ee6618c820d33" candidate_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8315d22c56f7439a9e1fc82f153f018c"
I20240501 07:59:16.140272 6445 raft_consensus.cc:3037] T 0b24f67cf1a64debb23ee6618c820d33 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:16.141139 8793 raft_consensus.cc:2454] T 0b24f67cf1a64debb23ee6618c820d33 P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b164cbbc00f4e8eb3254ddd349a3ff9 in term 1.
I20240501 07:59:16.141296 6157 catalog_manager.cc:4830] TS 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129:46875): tablet e3539c15a7bb4a09a967ad6b6b9601b2 (table rebalancer_dynamic_table_new_name [id=7ed9e34cd6994a18a4abe8cbb9b73232]) successfully deleted
I20240501 07:59:16.141351 6445 raft_consensus.cc:2454] T 0b24f67cf1a64debb23ee6618c820d33 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b164cbbc00f4e8eb3254ddd349a3ff9 in term 1.
I20240501 07:59:16.141500 6510 leader_election.cc:302] T 0b24f67cf1a64debb23ee6618c820d33 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b164cbbc00f4e8eb3254ddd349a3ff9, 74077b61d6f54342bf9736b4a9f2eade; no voters:
I20240501 07:59:16.141727 11107 raft_consensus.cc:2781] T 0b24f67cf1a64debb23ee6618c820d33 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:16.141822 11107 raft_consensus.cc:686] T 0b24f67cf1a64debb23ee6618c820d33 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 LEADER]: Becoming Leader. State: Replica: 3b164cbbc00f4e8eb3254ddd349a3ff9, State: Running, Role: LEADER
I20240501 07:59:16.141961 11107 consensus_queue.cc:237] T 0b24f67cf1a64debb23ee6618c820d33 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } }
I20240501 07:59:16.144724 6166 catalog_manager.cc:5484] T 0b24f67cf1a64debb23ee6618c820d33 P 3b164cbbc00f4e8eb3254ddd349a3ff9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } health_report { overall_health: UNKNOWN } } }
I20240501 07:59:16.158216 9188 consensus_queue.cc:237] T c5a876dfb598428ab350593129505504 P 83f20e84af4c40d09a4804b6b26e697f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: true } }
I20240501 07:59:16.159766 8793 raft_consensus.cc:1261] T c5a876dfb598428ab350593129505504 P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Refusing update from remote peer 83f20e84af4c40d09a4804b6b26e697f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 07:59:16.160133 9125 consensus_peers.cc:485] T c5a876dfb598428ab350593129505504 P 83f20e84af4c40d09a4804b6b26e697f -> Peer 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131:34471): Couldn't send request to peer 3b164cbbc00f4e8eb3254ddd349a3ff9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c5a876dfb598428ab350593129505504. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:16.160246 6445 raft_consensus.cc:1261] T c5a876dfb598428ab350593129505504 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Refusing update from remote peer 83f20e84af4c40d09a4804b6b26e697f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:16.160689 12838 consensus_queue.cc:1035] T c5a876dfb598428ab350593129505504 P 83f20e84af4c40d09a4804b6b26e697f [LEADER]: Connected to new peer: Peer: permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 07:59:16.160913 12503 consensus_queue.cc:1035] T c5a876dfb598428ab350593129505504 P 83f20e84af4c40d09a4804b6b26e697f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 07:59:16.162701 12649 raft_consensus.cc:2932] T c5a876dfb598428ab350593129505504 P 83f20e84af4c40d09a4804b6b26e697f [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: true } } }
I20240501 07:59:16.163115 8793 raft_consensus.cc:2932] T c5a876dfb598428ab350593129505504 P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: true } } }
I20240501 07:59:16.163115 6445 raft_consensus.cc:2932] T c5a876dfb598428ab350593129505504 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: true } } }
I20240501 07:59:16.166525 6165 catalog_manager.cc:5484] T c5a876dfb598428ab350593129505504 P 74077b61d6f54342bf9736b4a9f2eade reported cstate change: config changed from index -1 to 2, NON_VOTER 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131) added. New cstate: current_term: 1 leader_uuid: "83f20e84af4c40d09a4804b6b26e697f" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: true } } }
I20240501 07:59:16.175024 6575 raft_consensus.cc:1203] T 12ce6a21d06c4ba889542758be9dc62d P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 07:59:16.231904 6166 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55950:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 07:59:16.242798 9035 tablet_service.cc:1467] Processing CreateTablet for tablet a8e0e297022543f49ea7fb976c5c0606 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=c06f8c9068414fbd8aa85c90e8970596]), partition=
I20240501 07:59:16.243000 9167 tablet_service.cc:1467] Processing CreateTablet for tablet a8e0e297022543f49ea7fb976c5c0606 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=c06f8c9068414fbd8aa85c90e8970596]), partition=
I20240501 07:59:16.243120 9035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8e0e297022543f49ea7fb976c5c0606. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:16.243232 9167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8e0e297022543f49ea7fb976c5c0606. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:16.245365 12951 tablet_bootstrap.cc:492] T a8e0e297022543f49ea7fb976c5c0606 P a505e7cb133c4392bfda8a4a20874ae9: Bootstrap starting.
I20240501 07:59:16.246352 12951 tablet_bootstrap.cc:654] T a8e0e297022543f49ea7fb976c5c0606 P a505e7cb133c4392bfda8a4a20874ae9: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:16.246379 6556 tablet_service.cc:1467] Processing CreateTablet for tablet a8e0e297022543f49ea7fb976c5c0606 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=c06f8c9068414fbd8aa85c90e8970596]), partition=
I20240501 07:59:16.246620 6556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8e0e297022543f49ea7fb976c5c0606. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:16.248383 12951 tablet_bootstrap.cc:492] T a8e0e297022543f49ea7fb976c5c0606 P a505e7cb133c4392bfda8a4a20874ae9: No bootstrap required, opened a new log
I20240501 07:59:16.248481 12951 ts_tablet_manager.cc:1367] T a8e0e297022543f49ea7fb976c5c0606 P a505e7cb133c4392bfda8a4a20874ae9: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240501 07:59:16.248745 12949 tablet_bootstrap.cc:492] T a8e0e297022543f49ea7fb976c5c0606 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Bootstrap starting.
I20240501 07:59:16.249711 12949 tablet_bootstrap.cc:654] T a8e0e297022543f49ea7fb976c5c0606 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:16.249734 12951 raft_consensus.cc:348] T a8e0e297022543f49ea7fb976c5c0606 P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } }
I20240501 07:59:16.249868 12951 raft_consensus.cc:374] T a8e0e297022543f49ea7fb976c5c0606 P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:16.249898 12951 raft_consensus.cc:729] T a8e0e297022543f49ea7fb976c5c0606 P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a505e7cb133c4392bfda8a4a20874ae9, State: Initialized, Role: FOLLOWER
I20240501 07:59:16.250016 12951 consensus_queue.cc:260] T a8e0e297022543f49ea7fb976c5c0606 P a505e7cb133c4392bfda8a4a20874ae9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } }
I20240501 07:59:16.250276 12951 ts_tablet_manager.cc:1398] T a8e0e297022543f49ea7fb976c5c0606 P a505e7cb133c4392bfda8a4a20874ae9: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:59:16.249712 6575 consensus_queue.cc:237] T 114b6dbafa1c4c44ac42f1d2293e3125 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } }
I20240501 07:59:16.251968 12949 tablet_bootstrap.cc:492] T a8e0e297022543f49ea7fb976c5c0606 P 3b164cbbc00f4e8eb3254ddd349a3ff9: No bootstrap required, opened a new log
I20240501 07:59:16.252069 12949 ts_tablet_manager.cc:1367] T a8e0e297022543f49ea7fb976c5c0606 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 07:59:16.252092 13035 tablet_bootstrap.cc:492] T a8e0e297022543f49ea7fb976c5c0606 P 83f20e84af4c40d09a4804b6b26e697f: Bootstrap starting.
I20240501 07:59:16.252506 12949 raft_consensus.cc:348] T a8e0e297022543f49ea7fb976c5c0606 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } }
I20240501 07:59:16.252614 12949 raft_consensus.cc:374] T a8e0e297022543f49ea7fb976c5c0606 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:16.252651 12949 raft_consensus.cc:729] T a8e0e297022543f49ea7fb976c5c0606 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b164cbbc00f4e8eb3254ddd349a3ff9, State: Initialized, Role: FOLLOWER
I20240501 07:59:16.252753 12949 consensus_queue.cc:260] T a8e0e297022543f49ea7fb976c5c0606 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } }
I20240501 07:59:16.253062 13035 tablet_bootstrap.cc:654] T a8e0e297022543f49ea7fb976c5c0606 P 83f20e84af4c40d09a4804b6b26e697f: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:16.253084 12949 ts_tablet_manager.cc:1398] T a8e0e297022543f49ea7fb976c5c0606 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:16.253448 9056 raft_consensus.cc:1261] T 114b6dbafa1c4c44ac42f1d2293e3125 P a505e7cb133c4392bfda8a4a20874ae9 [term 1 FOLLOWER]: Refusing update from remote peer 3b164cbbc00f4e8eb3254ddd349a3ff9: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 07:59:16.254060 8793 raft_consensus.cc:1261] T 114b6dbafa1c4c44ac42f1d2293e3125 P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Refusing update from remote peer 3b164cbbc00f4e8eb3254ddd349a3ff9: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 07:59:16.254660 12119 consensus_queue.cc:1035] T 114b6dbafa1c4c44ac42f1d2293e3125 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 07:59:16.254915 10046 consensus_queue.cc:1035] T 114b6dbafa1c4c44ac42f1d2293e3125 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 07:59:16.255836 12646 raft_consensus.cc:1050] T 59e171892fc1414fbcfd6a8c090854f3 P 13d05ebdca8c4f489c597ce550dd773d: attempting to promote NON_VOTER 3b164cbbc00f4e8eb3254ddd349a3ff9 to VOTER
I20240501 07:59:16.256301 12646 consensus_queue.cc:237] T 59e171892fc1414fbcfd6a8c090854f3 P 13d05ebdca8c4f489c597ce550dd773d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { replace: true } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: false } }
I20240501 07:59:16.256793 12119 raft_consensus.cc:2932] T 114b6dbafa1c4c44ac42f1d2293e3125 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } } }
I20240501 07:59:16.257230 8793 raft_consensus.cc:2932] T 114b6dbafa1c4c44ac42f1d2293e3125 P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } } }
I20240501 07:59:16.257889 9188 raft_consensus.cc:1261] T 59e171892fc1414fbcfd6a8c090854f3 P 83f20e84af4c40d09a4804b6b26e697f [term 1 FOLLOWER]: Refusing update from remote peer 13d05ebdca8c4f489c597ce550dd773d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 07:59:16.258306 10478 consensus_queue.cc:1035] T 59e171892fc1414fbcfd6a8c090854f3 P 13d05ebdca8c4f489c597ce550dd773d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 07:59:16.259038 6160 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 114b6dbafa1c4c44ac42f1d2293e3125 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 07:59:16.260766 6166 catalog_manager.cc:5484] T 114b6dbafa1c4c44ac42f1d2293e3125 P 3b164cbbc00f4e8eb3254ddd349a3ff9 reported cstate change: config changed from index 8 to 9, VOTER 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 07:59:16.261018 13035 tablet_bootstrap.cc:492] T a8e0e297022543f49ea7fb976c5c0606 P 83f20e84af4c40d09a4804b6b26e697f: No bootstrap required, opened a new log
I20240501 07:59:16.261271 9056 raft_consensus.cc:2932] T 114b6dbafa1c4c44ac42f1d2293e3125 P a505e7cb133c4392bfda8a4a20874ae9 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } } }
I20240501 07:59:16.261694 13035 ts_tablet_manager.cc:1367] T a8e0e297022543f49ea7fb976c5c0606 P 83f20e84af4c40d09a4804b6b26e697f: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.002s
I20240501 07:59:16.262307 13035 raft_consensus.cc:348] T a8e0e297022543f49ea7fb976c5c0606 P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } }
I20240501 07:59:16.262907 6575 raft_consensus.cc:1261] T 59e171892fc1414fbcfd6a8c090854f3 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 LEARNER]: Refusing update from remote peer 13d05ebdca8c4f489c597ce550dd773d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 07:59:16.263042 13035 raft_consensus.cc:374] T a8e0e297022543f49ea7fb976c5c0606 P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:16.263136 13035 raft_consensus.cc:729] T a8e0e297022543f49ea7fb976c5c0606 P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83f20e84af4c40d09a4804b6b26e697f, State: Initialized, Role: FOLLOWER
I20240501 07:59:16.263306 13035 consensus_queue.cc:260] T a8e0e297022543f49ea7fb976c5c0606 P 83f20e84af4c40d09a4804b6b26e697f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } }
I20240501 07:59:16.263366 10478 consensus_queue.cc:1035] T 59e171892fc1414fbcfd6a8c090854f3 P 13d05ebdca8c4f489c597ce550dd773d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 07:59:16.264717 8793 raft_consensus.cc:1261] T 59e171892fc1414fbcfd6a8c090854f3 P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Refusing update from remote peer 13d05ebdca8c4f489c597ce550dd773d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 07:59:16.265347 13040 consensus_queue.cc:1035] T 59e171892fc1414fbcfd6a8c090854f3 P 13d05ebdca8c4f489c597ce550dd773d [LEADER]: Connected to new peer: Peer: permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 07:59:16.265985 13035 ts_tablet_manager.cc:1398] T a8e0e297022543f49ea7fb976c5c0606 P 83f20e84af4c40d09a4804b6b26e697f: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240501 07:59:16.266041 6424 tablet_service.cc:1514] Processing DeleteTablet for tablet 114b6dbafa1c4c44ac42f1d2293e3125 with delete_type TABLET_DATA_TOMBSTONED (TS 8315d22c56f7439a9e1fc82f153f018c not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:45034
I20240501 07:59:16.266280 12858 tablet_replica.cc:330] T 114b6dbafa1c4c44ac42f1d2293e3125 P 8315d22c56f7439a9e1fc82f153f018c: stopping tablet replica
I20240501 07:59:16.266431 12858 raft_consensus.cc:2229] T 114b6dbafa1c4c44ac42f1d2293e3125 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:16.266629 12858 raft_consensus.cc:2258] T 114b6dbafa1c4c44ac42f1d2293e3125 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:16.267972 12858 ts_tablet_manager.cc:1875] T 114b6dbafa1c4c44ac42f1d2293e3125 P 8315d22c56f7439a9e1fc82f153f018c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:59:16.270453 12858 ts_tablet_manager.cc:1888] T 114b6dbafa1c4c44ac42f1d2293e3125 P 8315d22c56f7439a9e1fc82f153f018c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240501 07:59:16.270529 12858 log.cc:1196] T 114b6dbafa1c4c44ac42f1d2293e3125 P 8315d22c56f7439a9e1fc82f153f018c: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/114b6dbafa1c4c44ac42f1d2293e3125
I20240501 07:59:16.271047 6159 catalog_manager.cc:4830] TS 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130:32941): tablet 114b6dbafa1c4c44ac42f1d2293e3125 (table rebalance_test_table_5 [id=336b9f044a794b6a8a51d76e720b0ecd]) successfully deleted
I20240501 07:59:16.271662 10478 raft_consensus.cc:2932] T 59e171892fc1414fbcfd6a8c090854f3 P 13d05ebdca8c4f489c597ce550dd773d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { replace: true } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: false } } }
I20240501 07:59:16.272269 6575 raft_consensus.cc:2932] T 59e171892fc1414fbcfd6a8c090854f3 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { replace: true } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: false } } }
I20240501 07:59:16.272624 8793 raft_consensus.cc:2932] T 59e171892fc1414fbcfd6a8c090854f3 P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { replace: true } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: false } } }
I20240501 07:59:16.274271 9188 raft_consensus.cc:2932] T 59e171892fc1414fbcfd6a8c090854f3 P 83f20e84af4c40d09a4804b6b26e697f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { replace: true } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: false } } }
I20240501 07:59:16.276542 6164 catalog_manager.cc:5484] T 59e171892fc1414fbcfd6a8c090854f3 P 13d05ebdca8c4f489c597ce550dd773d reported cstate change: config changed from index 2 to 3, 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "13d05ebdca8c4f489c597ce550dd773d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 07:59:16.279742 8925 consensus_queue.cc:237] T 59e171892fc1414fbcfd6a8c090854f3 P 13d05ebdca8c4f489c597ce550dd773d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: false } }
I20240501 07:59:16.280789 6575 raft_consensus.cc:1261] T 59e171892fc1414fbcfd6a8c090854f3 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Refusing update from remote peer 13d05ebdca8c4f489c597ce550dd773d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:59:16.281203 13040 consensus_queue.cc:1035] T 59e171892fc1414fbcfd6a8c090854f3 P 13d05ebdca8c4f489c597ce550dd773d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:16.281284 8793 raft_consensus.cc:1261] T 59e171892fc1414fbcfd6a8c090854f3 P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Refusing update from remote peer 13d05ebdca8c4f489c597ce550dd773d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:59:16.281637 13040 consensus_queue.cc:1035] T 59e171892fc1414fbcfd6a8c090854f3 P 13d05ebdca8c4f489c597ce550dd773d [LEADER]: Connected to new peer: Peer: permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:16.282763 12840 raft_consensus.cc:2932] T 59e171892fc1414fbcfd6a8c090854f3 P 13d05ebdca8c4f489c597ce550dd773d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: false } } }
I20240501 07:59:16.284010 8793 raft_consensus.cc:2932] T 59e171892fc1414fbcfd6a8c090854f3 P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: false } } }
I20240501 07:59:16.284353 6159 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 59e171892fc1414fbcfd6a8c090854f3 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 07:59:16.285173 6575 raft_consensus.cc:2932] T 59e171892fc1414fbcfd6a8c090854f3 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: false } } }
I20240501 07:59:16.285884 6170 catalog_manager.cc:5484] T 59e171892fc1414fbcfd6a8c090854f3 P 13d05ebdca8c4f489c597ce550dd773d reported cstate change: config changed from index 3 to 4, VOTER 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136) evicted. New cstate: current_term: 1 leader_uuid: "13d05ebdca8c4f489c597ce550dd773d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 07:59:16.289009 9167 tablet_service.cc:1514] Processing DeleteTablet for tablet 59e171892fc1414fbcfd6a8c090854f3 with delete_type TABLET_DATA_TOMBSTONED (TS 83f20e84af4c40d09a4804b6b26e697f not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:33594
I20240501 07:59:16.289402 13044 tablet_replica.cc:330] T 59e171892fc1414fbcfd6a8c090854f3 P 83f20e84af4c40d09a4804b6b26e697f: stopping tablet replica
I20240501 07:59:16.289577 13044 raft_consensus.cc:2229] T 59e171892fc1414fbcfd6a8c090854f3 P 83f20e84af4c40d09a4804b6b26e697f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:16.289752 13044 raft_consensus.cc:2258] T 59e171892fc1414fbcfd6a8c090854f3 P 83f20e84af4c40d09a4804b6b26e697f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:16.290285 13044 ts_tablet_manager.cc:1875] T 59e171892fc1414fbcfd6a8c090854f3 P 83f20e84af4c40d09a4804b6b26e697f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:59:16.292385 13044 ts_tablet_manager.cc:1888] T 59e171892fc1414fbcfd6a8c090854f3 P 83f20e84af4c40d09a4804b6b26e697f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 07:59:16.292457 13044 log.cc:1196] T 59e171892fc1414fbcfd6a8c090854f3 P 83f20e84af4c40d09a4804b6b26e697f: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/59e171892fc1414fbcfd6a8c090854f3
I20240501 07:59:16.292991 6158 catalog_manager.cc:4830] TS 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): tablet 59e171892fc1414fbcfd6a8c090854f3 (table rebalancer_extra_table_12 [id=731fe27ae8e34d92bd595d43c4bc8f3a]) successfully deleted
I20240501 07:59:16.293287 12048 raft_consensus.cc:1050] T 6221deb431d94bfca1c14696e0e0a0c5 P 1f9cb72be98941b5a8e504cbe34132b9: attempting to promote NON_VOTER 74077b61d6f54342bf9736b4a9f2eade to VOTER
I20240501 07:59:16.293642 12048 consensus_queue.cc:237] T 6221deb431d94bfca1c14696e0e0a0c5 P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } }
I20240501 07:59:16.294916 6575 raft_consensus.cc:1261] T 6221deb431d94bfca1c14696e0e0a0c5 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Refusing update from remote peer 1f9cb72be98941b5a8e504cbe34132b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:59:16.294939 8793 raft_consensus.cc:1261] T 6221deb431d94bfca1c14696e0e0a0c5 P 74077b61d6f54342bf9736b4a9f2eade [term 1 LEARNER]: Refusing update from remote peer 1f9cb72be98941b5a8e504cbe34132b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:59:16.295035 6445 raft_consensus.cc:1261] T 6221deb431d94bfca1c14696e0e0a0c5 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Refusing update from remote peer 1f9cb72be98941b5a8e504cbe34132b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:59:16.295354 13046 consensus_queue.cc:1035] T 6221deb431d94bfca1c14696e0e0a0c5 P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:59:16.295562 12048 consensus_queue.cc:1035] T 6221deb431d94bfca1c14696e0e0a0c5 P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:59:16.295713 11719 consensus_queue.cc:1035] T 6221deb431d94bfca1c14696e0e0a0c5 P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:59:16.297783 11719 raft_consensus.cc:2932] T 6221deb431d94bfca1c14696e0e0a0c5 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } } }
I20240501 07:59:16.298439 8793 raft_consensus.cc:2932] T 6221deb431d94bfca1c14696e0e0a0c5 P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } } }
I20240501 07:59:16.299044 6575 raft_consensus.cc:2932] T 6221deb431d94bfca1c14696e0e0a0c5 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } } }
I20240501 07:59:16.301463 6165 catalog_manager.cc:5484] T 6221deb431d94bfca1c14696e0e0a0c5 P 1f9cb72be98941b5a8e504cbe34132b9 reported cstate change: config changed from index 4 to 5, 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1f9cb72be98941b5a8e504cbe34132b9" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 07:59:16.301656 6445 raft_consensus.cc:2932] T 6221deb431d94bfca1c14696e0e0a0c5 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } } }
I20240501 07:59:16.304701 6707 consensus_queue.cc:237] T 6221deb431d94bfca1c14696e0e0a0c5 P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } }
I20240501 07:59:16.306231 8793 raft_consensus.cc:1261] T 6221deb431d94bfca1c14696e0e0a0c5 P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Refusing update from remote peer 1f9cb72be98941b5a8e504cbe34132b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 07:59:16.306385 6575 raft_consensus.cc:1261] T 6221deb431d94bfca1c14696e0e0a0c5 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Refusing update from remote peer 1f9cb72be98941b5a8e504cbe34132b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 07:59:16.306656 13046 consensus_queue.cc:1035] T 6221deb431d94bfca1c14696e0e0a0c5 P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 07:59:16.306835 11402 consensus_queue.cc:1035] T 6221deb431d94bfca1c14696e0e0a0c5 P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 07:59:16.309082 11402 raft_consensus.cc:2932] T 6221deb431d94bfca1c14696e0e0a0c5 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } } }
I20240501 07:59:16.309511 6575 raft_consensus.cc:2932] T 6221deb431d94bfca1c14696e0e0a0c5 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } } }
I20240501 07:59:16.310714 6159 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6221deb431d94bfca1c14696e0e0a0c5 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 07:59:16.312301 12488 raft_consensus.cc:482] T c16b7962c18d4e02b850c7c7c678eae4 P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:16.312386 12488 raft_consensus.cc:504] T c16b7962c18d4e02b850c7c7c678eae4 P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } }
I20240501 07:59:16.312167 8793 raft_consensus.cc:2932] T 6221deb431d94bfca1c14696e0e0a0c5 P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } } }
I20240501 07:59:16.312783 12488 leader_election.cc:288] T c16b7962c18d4e02b850c7c7c678eae4 P a505e7cb133c4392bfda8a4a20874ae9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133:39859), 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134:40141)
I20240501 07:59:16.313041 8794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c16b7962c18d4e02b850c7c7c678eae4" candidate_uuid: "a505e7cb133c4392bfda8a4a20874ae9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74077b61d6f54342bf9736b4a9f2eade" is_pre_election: true
I20240501 07:59:16.313160 8794 raft_consensus.cc:2454] T c16b7962c18d4e02b850c7c7c678eae4 P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a505e7cb133c4392bfda8a4a20874ae9 in term 0.
I20240501 07:59:16.313314 8925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c16b7962c18d4e02b850c7c7c678eae4" candidate_uuid: "a505e7cb133c4392bfda8a4a20874ae9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13d05ebdca8c4f489c597ce550dd773d" is_pre_election: true
I20240501 07:59:16.313432 8989 leader_election.cc:302] T c16b7962c18d4e02b850c7c7c678eae4 P a505e7cb133c4392bfda8a4a20874ae9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74077b61d6f54342bf9736b4a9f2eade, a505e7cb133c4392bfda8a4a20874ae9; no voters:
I20240501 07:59:16.313445 8925 raft_consensus.cc:2454] T c16b7962c18d4e02b850c7c7c678eae4 P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a505e7cb133c4392bfda8a4a20874ae9 in term 0.
I20240501 07:59:16.313599 12488 raft_consensus.cc:2781] T c16b7962c18d4e02b850c7c7c678eae4 P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:16.313650 12488 raft_consensus.cc:482] T c16b7962c18d4e02b850c7c7c678eae4 P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:16.313522 6170 catalog_manager.cc:5484] T 6221deb431d94bfca1c14696e0e0a0c5 P 3b164cbbc00f4e8eb3254ddd349a3ff9 reported cstate change: config changed from index 5 to 6, VOTER 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "1f9cb72be98941b5a8e504cbe34132b9" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } } }
I20240501 07:59:16.313696 12488 raft_consensus.cc:3037] T c16b7962c18d4e02b850c7c7c678eae4 P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:16.314675 12488 raft_consensus.cc:504] T c16b7962c18d4e02b850c7c7c678eae4 P a505e7cb133c4392bfda8a4a20874ae9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } }
I20240501 07:59:16.315029 12488 leader_election.cc:288] T c16b7962c18d4e02b850c7c7c678eae4 P a505e7cb133c4392bfda8a4a20874ae9 [CANDIDATE]: Term 1 election: Requested vote from peers 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133:39859), 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134:40141)
I20240501 07:59:16.315299 8794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c16b7962c18d4e02b850c7c7c678eae4" candidate_uuid: "a505e7cb133c4392bfda8a4a20874ae9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74077b61d6f54342bf9736b4a9f2eade"
I20240501 07:59:16.315402 8794 raft_consensus.cc:3037] T c16b7962c18d4e02b850c7c7c678eae4 P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:16.315615 8925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c16b7962c18d4e02b850c7c7c678eae4" candidate_uuid: "a505e7cb133c4392bfda8a4a20874ae9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13d05ebdca8c4f489c597ce550dd773d"
I20240501 07:59:16.315719 8925 raft_consensus.cc:3037] T c16b7962c18d4e02b850c7c7c678eae4 P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:16.316463 8794 raft_consensus.cc:2454] T c16b7962c18d4e02b850c7c7c678eae4 P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a505e7cb133c4392bfda8a4a20874ae9 in term 1.
I20240501 07:59:16.316741 8925 raft_consensus.cc:2454] T c16b7962c18d4e02b850c7c7c678eae4 P 13d05ebdca8c4f489c597ce550dd773d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a505e7cb133c4392bfda8a4a20874ae9 in term 1.
I20240501 07:59:16.316764 13046 raft_consensus.cc:482] T 62936b541ca541faa99eb36bf24c6404 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:16.316839 8989 leader_election.cc:302] T c16b7962c18d4e02b850c7c7c678eae4 P a505e7cb133c4392bfda8a4a20874ae9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74077b61d6f54342bf9736b4a9f2eade, a505e7cb133c4392bfda8a4a20874ae9; no voters:
I20240501 07:59:16.316857 13046 raft_consensus.cc:504] T 62936b541ca541faa99eb36bf24c6404 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:59:16.317471 12488 raft_consensus.cc:2781] T c16b7962c18d4e02b850c7c7c678eae4 P a505e7cb133c4392bfda8a4a20874ae9 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:16.317579 12488 raft_consensus.cc:686] T c16b7962c18d4e02b850c7c7c678eae4 P a505e7cb133c4392bfda8a4a20874ae9 [term 1 LEADER]: Becoming Leader. State: Replica: a505e7cb133c4392bfda8a4a20874ae9, State: Running, Role: LEADER
I20240501 07:59:16.317719 12488 consensus_queue.cc:237] T c16b7962c18d4e02b850c7c7c678eae4 P a505e7cb133c4392bfda8a4a20874ae9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } }
I20240501 07:59:16.317926 13046 leader_election.cc:288] T 62936b541ca541faa99eb36bf24c6404 P 1f9cb72be98941b5a8e504cbe34132b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134:40141), 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130:32941)
I20240501 07:59:16.318377 8925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62936b541ca541faa99eb36bf24c6404" candidate_uuid: "1f9cb72be98941b5a8e504cbe34132b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13d05ebdca8c4f489c597ce550dd773d" is_pre_election: true
I20240501 07:59:16.318485 8925 raft_consensus.cc:2454] T 62936b541ca541faa99eb36bf24c6404 P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f9cb72be98941b5a8e504cbe34132b9 in term 0.
I20240501 07:59:16.318737 6643 leader_election.cc:302] T 62936b541ca541faa99eb36bf24c6404 P 1f9cb72be98941b5a8e504cbe34132b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13d05ebdca8c4f489c597ce550dd773d, 1f9cb72be98941b5a8e504cbe34132b9; no voters:
I20240501 07:59:16.318903 11402 raft_consensus.cc:2781] T 62936b541ca541faa99eb36bf24c6404 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:16.318954 11402 raft_consensus.cc:482] T 62936b541ca541faa99eb36bf24c6404 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:16.319001 11402 raft_consensus.cc:3037] T 62936b541ca541faa99eb36bf24c6404 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:16.318974 6445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62936b541ca541faa99eb36bf24c6404" candidate_uuid: "1f9cb72be98941b5a8e504cbe34132b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8315d22c56f7439a9e1fc82f153f018c" is_pre_election: true
I20240501 07:59:16.319111 6445 raft_consensus.cc:2454] T 62936b541ca541faa99eb36bf24c6404 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f9cb72be98941b5a8e504cbe34132b9 in term 0.
I20240501 07:59:16.320225 6424 tablet_service.cc:1514] Processing DeleteTablet for tablet 6221deb431d94bfca1c14696e0e0a0c5 with delete_type TABLET_DATA_TOMBSTONED (TS 8315d22c56f7439a9e1fc82f153f018c not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:45034
I20240501 07:59:16.320453 12858 tablet_replica.cc:330] T 6221deb431d94bfca1c14696e0e0a0c5 P 8315d22c56f7439a9e1fc82f153f018c: stopping tablet replica
I20240501 07:59:16.320480 11402 raft_consensus.cc:504] T 62936b541ca541faa99eb36bf24c6404 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:59:16.320622 12858 raft_consensus.cc:2229] T 6221deb431d94bfca1c14696e0e0a0c5 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:16.320756 12858 raft_consensus.cc:2258] T 6221deb431d94bfca1c14696e0e0a0c5 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:16.320828 11402 leader_election.cc:288] T 62936b541ca541faa99eb36bf24c6404 P 1f9cb72be98941b5a8e504cbe34132b9 [CANDIDATE]: Term 1 election: Requested vote from peers 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134:40141), 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130:32941)
I20240501 07:59:16.321143 8925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62936b541ca541faa99eb36bf24c6404" candidate_uuid: "1f9cb72be98941b5a8e504cbe34132b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13d05ebdca8c4f489c597ce550dd773d"
I20240501 07:59:16.321177 6445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62936b541ca541faa99eb36bf24c6404" candidate_uuid: "1f9cb72be98941b5a8e504cbe34132b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8315d22c56f7439a9e1fc82f153f018c"
I20240501 07:59:16.321257 8925 raft_consensus.cc:3037] T 62936b541ca541faa99eb36bf24c6404 P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:16.321292 6445 raft_consensus.cc:3037] T 62936b541ca541faa99eb36bf24c6404 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:16.321575 6166 catalog_manager.cc:5484] T c16b7962c18d4e02b850c7c7c678eae4 P a505e7cb133c4392bfda8a4a20874ae9 reported cstate change: term changed from 0 to 1, leader changed from <none> to a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135). New cstate: current_term: 1 leader_uuid: "a505e7cb133c4392bfda8a4a20874ae9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } health_report { overall_health: HEALTHY } } }
I20240501 07:59:16.322032 8925 raft_consensus.cc:2454] T 62936b541ca541faa99eb36bf24c6404 P 13d05ebdca8c4f489c597ce550dd773d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f9cb72be98941b5a8e504cbe34132b9 in term 1.
I20240501 07:59:16.322314 6445 raft_consensus.cc:2454] T 62936b541ca541faa99eb36bf24c6404 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f9cb72be98941b5a8e504cbe34132b9 in term 1.
I20240501 07:59:16.322590 6643 leader_election.cc:302] T 62936b541ca541faa99eb36bf24c6404 P 1f9cb72be98941b5a8e504cbe34132b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f9cb72be98941b5a8e504cbe34132b9, 8315d22c56f7439a9e1fc82f153f018c; no voters:
I20240501 07:59:16.322991 11402 raft_consensus.cc:2781] T 62936b541ca541faa99eb36bf24c6404 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:16.323076 11402 raft_consensus.cc:686] T 62936b541ca541faa99eb36bf24c6404 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 LEADER]: Becoming Leader. State: Replica: 1f9cb72be98941b5a8e504cbe34132b9, State: Running, Role: LEADER
I20240501 07:59:16.323195 11402 consensus_queue.cc:237] T 62936b541ca541faa99eb36bf24c6404 P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:59:16.323237 12858 ts_tablet_manager.cc:1875] T 6221deb431d94bfca1c14696e0e0a0c5 P 8315d22c56f7439a9e1fc82f153f018c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:59:16.325834 9188 raft_consensus.cc:1203] T b7d57b7fd5234f50afbc859bc997f3e1 P 83f20e84af4c40d09a4804b6b26e697f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240501 07:59:16.325884 6166 catalog_manager.cc:5484] T 62936b541ca541faa99eb36bf24c6404 P 1f9cb72be98941b5a8e504cbe34132b9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132). New cstate: current_term: 1 leader_uuid: "1f9cb72be98941b5a8e504cbe34132b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } health_report { overall_health: HEALTHY } } }
I20240501 07:59:16.326567 12858 ts_tablet_manager.cc:1888] T 6221deb431d94bfca1c14696e0e0a0c5 P 8315d22c56f7439a9e1fc82f153f018c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 07:59:16.326704 12858 log.cc:1196] T 6221deb431d94bfca1c14696e0e0a0c5 P 8315d22c56f7439a9e1fc82f153f018c: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/6221deb431d94bfca1c14696e0e0a0c5
I20240501 07:59:16.327466 6159 catalog_manager.cc:4830] TS 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130:32941): tablet 6221deb431d94bfca1c14696e0e0a0c5 (table rebalance_test_table_13 [id=886e8147658c4cffb480ca3fb821c563]) successfully deleted
I20240501 07:59:16.333124 12488 raft_consensus.cc:482] T a8e0e297022543f49ea7fb976c5c0606 P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:16.333263 12488 raft_consensus.cc:504] T a8e0e297022543f49ea7fb976c5c0606 P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } }
I20240501 07:59:16.333663 12488 leader_election.cc:288] T a8e0e297022543f49ea7fb976c5c0606 P a505e7cb133c4392bfda8a4a20874ae9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149), 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131:34471)
I20240501 07:59:16.334050 6575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8e0e297022543f49ea7fb976c5c0606" candidate_uuid: "a505e7cb133c4392bfda8a4a20874ae9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" is_pre_election: true
I20240501 07:59:16.334189 9188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8e0e297022543f49ea7fb976c5c0606" candidate_uuid: "a505e7cb133c4392bfda8a4a20874ae9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83f20e84af4c40d09a4804b6b26e697f" is_pre_election: true
I20240501 07:59:16.334367 6575 raft_consensus.cc:2454] T a8e0e297022543f49ea7fb976c5c0606 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a505e7cb133c4392bfda8a4a20874ae9 in term 0.
I20240501 07:59:16.334373 9188 raft_consensus.cc:2454] T a8e0e297022543f49ea7fb976c5c0606 P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a505e7cb133c4392bfda8a4a20874ae9 in term 0.
I20240501 07:59:16.334652 8992 leader_election.cc:302] T a8e0e297022543f49ea7fb976c5c0606 P a505e7cb133c4392bfda8a4a20874ae9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b164cbbc00f4e8eb3254ddd349a3ff9, a505e7cb133c4392bfda8a4a20874ae9; no voters:
I20240501 07:59:16.334846 12488 raft_consensus.cc:2781] T a8e0e297022543f49ea7fb976c5c0606 P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:16.334910 12488 raft_consensus.cc:482] T a8e0e297022543f49ea7fb976c5c0606 P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:16.334949 12488 raft_consensus.cc:3037] T a8e0e297022543f49ea7fb976c5c0606 P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:16.335752 12488 raft_consensus.cc:504] T a8e0e297022543f49ea7fb976c5c0606 P a505e7cb133c4392bfda8a4a20874ae9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } }
I20240501 07:59:16.336091 12488 leader_election.cc:288] T a8e0e297022543f49ea7fb976c5c0606 P a505e7cb133c4392bfda8a4a20874ae9 [CANDIDATE]: Term 1 election: Requested vote from peers 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149), 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131:34471)
I20240501 07:59:16.336421 6575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8e0e297022543f49ea7fb976c5c0606" candidate_uuid: "a505e7cb133c4392bfda8a4a20874ae9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9"
I20240501 07:59:16.336457 9188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8e0e297022543f49ea7fb976c5c0606" candidate_uuid: "a505e7cb133c4392bfda8a4a20874ae9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83f20e84af4c40d09a4804b6b26e697f"
I20240501 07:59:16.336527 6575 raft_consensus.cc:3037] T a8e0e297022543f49ea7fb976c5c0606 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:16.336580 9188 raft_consensus.cc:3037] T a8e0e297022543f49ea7fb976c5c0606 P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:16.337512 9188 raft_consensus.cc:2454] T a8e0e297022543f49ea7fb976c5c0606 P 83f20e84af4c40d09a4804b6b26e697f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a505e7cb133c4392bfda8a4a20874ae9 in term 1.
I20240501 07:59:16.337543 6575 raft_consensus.cc:2454] T a8e0e297022543f49ea7fb976c5c0606 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a505e7cb133c4392bfda8a4a20874ae9 in term 1.
I20240501 07:59:16.337855 8992 leader_election.cc:302] T a8e0e297022543f49ea7fb976c5c0606 P a505e7cb133c4392bfda8a4a20874ae9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b164cbbc00f4e8eb3254ddd349a3ff9, a505e7cb133c4392bfda8a4a20874ae9; no voters:
I20240501 07:59:16.338033 12488 raft_consensus.cc:2781] T a8e0e297022543f49ea7fb976c5c0606 P a505e7cb133c4392bfda8a4a20874ae9 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:16.338121 12488 raft_consensus.cc:686] T a8e0e297022543f49ea7fb976c5c0606 P a505e7cb133c4392bfda8a4a20874ae9 [term 1 LEADER]: Becoming Leader. State: Replica: a505e7cb133c4392bfda8a4a20874ae9, State: Running, Role: LEADER
I20240501 07:59:16.338263 12488 consensus_queue.cc:237] T a8e0e297022543f49ea7fb976c5c0606 P a505e7cb133c4392bfda8a4a20874ae9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } }
I20240501 07:59:16.340867 6166 catalog_manager.cc:5484] T a8e0e297022543f49ea7fb976c5c0606 P a505e7cb133c4392bfda8a4a20874ae9 reported cstate change: term changed from 0 to 1, leader changed from <none> to a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135). New cstate: current_term: 1 leader_uuid: "a505e7cb133c4392bfda8a4a20874ae9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } health_report { overall_health: UNKNOWN } } }
I20240501 07:59:16.450498 6166 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55950:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 07:59:16.455070 6575 raft_consensus.cc:1261] T a8e0e297022543f49ea7fb976c5c0606 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Refusing update from remote peer a505e7cb133c4392bfda8a4a20874ae9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:16.455153 9188 raft_consensus.cc:1261] T a8e0e297022543f49ea7fb976c5c0606 P 83f20e84af4c40d09a4804b6b26e697f [term 1 FOLLOWER]: Refusing update from remote peer a505e7cb133c4392bfda8a4a20874ae9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:16.455595 12488 consensus_queue.cc:1035] T a8e0e297022543f49ea7fb976c5c0606 P a505e7cb133c4392bfda8a4a20874ae9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:16.455900 12854 consensus_queue.cc:1035] T a8e0e297022543f49ea7fb976c5c0606 P a505e7cb133c4392bfda8a4a20874ae9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:16.458652 6166 catalog_manager.cc:5576] TS 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131:34471) does not have the latest schema for tablet a8e0e297022543f49ea7fb976c5c0606 (table rebalancer_dynamic_table [id=c06f8c9068414fbd8aa85c90e8970596]). Expected version 1 got 0
I20240501 07:59:16.459789 6165 catalog_manager.cc:5576] TS 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149) does not have the latest schema for tablet a8e0e297022543f49ea7fb976c5c0606 (table rebalancer_dynamic_table [id=c06f8c9068414fbd8aa85c90e8970596]). Expected version 1 got 0
I20240501 07:59:16.459841 12850 tablet.cc:1722] T a8e0e297022543f49ea7fb976c5c0606 P a505e7cb133c4392bfda8a4a20874ae9: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 07:59:16.460202 12850 tablet.cc:1620] T a8e0e297022543f49ea7fb976c5c0606 P a505e7cb133c4392bfda8a4a20874ae9: MemRowSet was empty: no flush needed.
I20240501 07:59:16.462714 6165 catalog_manager.cc:5576] TS 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131:34471) does not have the latest schema for tablet a8e0e297022543f49ea7fb976c5c0606 (table rebalancer_dynamic_table [id=c06f8c9068414fbd8aa85c90e8970596]). Expected version 1 got 0
I20240501 07:59:16.463994 6165 catalog_manager.cc:5576] TS 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149) does not have the latest schema for tablet a8e0e297022543f49ea7fb976c5c0606 (table rebalancer_dynamic_table [id=c06f8c9068414fbd8aa85c90e8970596]). Expected version 1 got 0
I20240501 07:59:16.464903 13092 tablet.cc:1722] T a8e0e297022543f49ea7fb976c5c0606 P 83f20e84af4c40d09a4804b6b26e697f: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 07:59:16.465363 13092 tablet.cc:1620] T a8e0e297022543f49ea7fb976c5c0606 P 83f20e84af4c40d09a4804b6b26e697f: MemRowSet was empty: no flush needed.
I20240501 07:59:16.465395 13088 mvcc.cc:205] Tried to move back new op lower bound from 7022798260036284416 to 7022798259563307008. Current Snapshot: MvccSnapshot[applied={T|T < 7022798260036284416}]
I20240501 07:59:16.465718 6170 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 0bc706e85ce74cef97a4827ea4d4aa65: rebalancer_dynamic_table [id=c06f8c9068414fbd8aa85c90e8970596] alter complete (version 1)
I20240501 07:59:16.470335 13091 tablet.cc:1722] T a8e0e297022543f49ea7fb976c5c0606 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 07:59:16.470671 13091 tablet.cc:1620] T a8e0e297022543f49ea7fb976c5c0606 P 3b164cbbc00f4e8eb3254ddd349a3ff9: MemRowSet was empty: no flush needed.
I20240501 07:59:16.484318 6313 consensus_queue.cc:237] T 5ee738571af740c1b494d4f17413d1d8 P 8b8b3ad3aa8f4053b2167dd550f2086f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: true } }
I20240501 07:59:16.485885 8793 raft_consensus.cc:1261] T 5ee738571af740c1b494d4f17413d1d8 P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Refusing update from remote peer 8b8b3ad3aa8f4053b2167dd550f2086f: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 07:59:16.485885 6575 raft_consensus.cc:1261] T 5ee738571af740c1b494d4f17413d1d8 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Refusing update from remote peer 8b8b3ad3aa8f4053b2167dd550f2086f: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240501 07:59:16.486296 6249 consensus_peers.cc:485] T 5ee738571af740c1b494d4f17413d1d8 P 8b8b3ad3aa8f4053b2167dd550f2086f -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5ee738571af740c1b494d4f17413d1d8. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:16.486366 12535 consensus_queue.cc:1035] T 5ee738571af740c1b494d4f17413d1d8 P 8b8b3ad3aa8f4053b2167dd550f2086f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 07:59:16.486843 11901 consensus_queue.cc:1035] T 5ee738571af740c1b494d4f17413d1d8 P 8b8b3ad3aa8f4053b2167dd550f2086f [LEADER]: Connected to new peer: Peer: permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 07:59:16.488423 12159 raft_consensus.cc:2932] T 5ee738571af740c1b494d4f17413d1d8 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: true } } }
I20240501 07:59:16.488860 6575 raft_consensus.cc:2932] T 5ee738571af740c1b494d4f17413d1d8 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: true } } }
I20240501 07:59:16.489001 8793 raft_consensus.cc:2932] T 5ee738571af740c1b494d4f17413d1d8 P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: true } } }
I20240501 07:59:16.491947 6165 catalog_manager.cc:5484] T 5ee738571af740c1b494d4f17413d1d8 P 74077b61d6f54342bf9736b4a9f2eade reported cstate change: config changed from index 6 to 7, NON_VOTER 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: true } } }
I20240501 07:59:16.569240 6170 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55950:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 07:59:16.574764 12850 tablet.cc:1722] T a8e0e297022543f49ea7fb976c5c0606 P a505e7cb133c4392bfda8a4a20874ae9: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 07:59:16.575116 12850 tablet.cc:1620] T a8e0e297022543f49ea7fb976c5c0606 P a505e7cb133c4392bfda8a4a20874ae9: MemRowSet was empty: no flush needed.
I20240501 07:59:16.575295 13092 tablet.cc:1722] T a8e0e297022543f49ea7fb976c5c0606 P 83f20e84af4c40d09a4804b6b26e697f: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 07:59:16.575341 13091 tablet.cc:1722] T a8e0e297022543f49ea7fb976c5c0606 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 07:59:16.575582 13092 tablet.cc:1620] T a8e0e297022543f49ea7fb976c5c0606 P 83f20e84af4c40d09a4804b6b26e697f: MemRowSet was empty: no flush needed.
I20240501 07:59:16.575582 13091 tablet.cc:1620] T a8e0e297022543f49ea7fb976c5c0606 P 3b164cbbc00f4e8eb3254ddd349a3ff9: MemRowSet was empty: no flush needed.
I20240501 07:59:16.579414 13126 ts_tablet_manager.cc:902] T c3a9036ccfee45498bca9681518d3699 P 83f20e84af4c40d09a4804b6b26e697f: Initiating tablet copy from peer 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130:32941)
I20240501 07:59:16.580039 13126 tablet_copy_client.cc:308] T c3a9036ccfee45498bca9681518d3699 P 83f20e84af4c40d09a4804b6b26e697f: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:32941
I20240501 07:59:16.580329 6166 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 0bc706e85ce74cef97a4827ea4d4aa65: rebalancer_dynamic_table [id=c06f8c9068414fbd8aa85c90e8970596] alter complete (version 2)
I20240501 07:59:16.580533 6466 tablet_copy_service.cc:140] P 8315d22c56f7439a9e1fc82f153f018c: Received BeginTabletCopySession request for tablet c3a9036ccfee45498bca9681518d3699 from peer 83f20e84af4c40d09a4804b6b26e697f ({username='slave'} at 127.0.99.136:51357)
I20240501 07:59:16.580662 6466 tablet_copy_service.cc:161] P 8315d22c56f7439a9e1fc82f153f018c: Beginning new tablet copy session on tablet c3a9036ccfee45498bca9681518d3699 from peer 83f20e84af4c40d09a4804b6b26e697f at {username='slave'} at 127.0.99.136:51357: session id = 83f20e84af4c40d09a4804b6b26e697f-c3a9036ccfee45498bca9681518d3699
I20240501 07:59:16.581754 6466 tablet_copy_source_session.cc:205] T c3a9036ccfee45498bca9681518d3699 P 8315d22c56f7439a9e1fc82f153f018c: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:16.583107 13126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3a9036ccfee45498bca9681518d3699. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:16.585151 13126 tablet_copy_client.cc:783] T c3a9036ccfee45498bca9681518d3699 P 83f20e84af4c40d09a4804b6b26e697f: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:16.585444 13126 tablet_copy_client.cc:647] T c3a9036ccfee45498bca9681518d3699 P 83f20e84af4c40d09a4804b6b26e697f: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:16.586583 13126 tablet_copy_client.cc:523] T c3a9036ccfee45498bca9681518d3699 P 83f20e84af4c40d09a4804b6b26e697f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:16.587822 13126 tablet_bootstrap.cc:492] T c3a9036ccfee45498bca9681518d3699 P 83f20e84af4c40d09a4804b6b26e697f: Bootstrap starting.
I20240501 07:59:16.610524 13126 tablet_bootstrap.cc:492] T c3a9036ccfee45498bca9681518d3699 P 83f20e84af4c40d09a4804b6b26e697f: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:16.610888 13126 tablet_bootstrap.cc:492] T c3a9036ccfee45498bca9681518d3699 P 83f20e84af4c40d09a4804b6b26e697f: Bootstrap complete.
I20240501 07:59:16.611095 13126 ts_tablet_manager.cc:1367] T c3a9036ccfee45498bca9681518d3699 P 83f20e84af4c40d09a4804b6b26e697f: Time spent bootstrapping tablet: real 0.023s user 0.022s sys 0.000s
I20240501 07:59:16.611718 13126 raft_consensus.cc:348] T c3a9036ccfee45498bca9681518d3699 P 83f20e84af4c40d09a4804b6b26e697f [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: true } }
I20240501 07:59:16.611868 13126 raft_consensus.cc:729] T c3a9036ccfee45498bca9681518d3699 P 83f20e84af4c40d09a4804b6b26e697f [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 83f20e84af4c40d09a4804b6b26e697f, State: Initialized, Role: LEARNER
I20240501 07:59:16.611992 13126 consensus_queue.cc:260] T c3a9036ccfee45498bca9681518d3699 P 83f20e84af4c40d09a4804b6b26e697f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: true } }
I20240501 07:59:16.612291 13126 ts_tablet_manager.cc:1398] T c3a9036ccfee45498bca9681518d3699 P 83f20e84af4c40d09a4804b6b26e697f: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20240501 07:59:16.612854 6466 tablet_copy_service.cc:342] P 8315d22c56f7439a9e1fc82f153f018c: Request end of tablet copy session 83f20e84af4c40d09a4804b6b26e697f-c3a9036ccfee45498bca9681518d3699 received from {username='slave'} at 127.0.99.136:51357
I20240501 07:59:16.612946 6466 tablet_copy_service.cc:433] P 8315d22c56f7439a9e1fc82f153f018c: ending tablet copy session 83f20e84af4c40d09a4804b6b26e697f-c3a9036ccfee45498bca9681518d3699 on tablet c3a9036ccfee45498bca9681518d3699 with peer 83f20e84af4c40d09a4804b6b26e697f
I20240501 07:59:16.618566 13132 ts_tablet_manager.cc:902] T c5a876dfb598428ab350593129505504 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Initiating tablet copy from peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149)
I20240501 07:59:16.619063 13132 tablet_copy_client.cc:308] T c5a876dfb598428ab350593129505504 P 3b164cbbc00f4e8eb3254ddd349a3ff9: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.136:46149
I20240501 07:59:16.619519 9207 tablet_copy_service.cc:140] P 83f20e84af4c40d09a4804b6b26e697f: Received BeginTabletCopySession request for tablet c5a876dfb598428ab350593129505504 from peer 3b164cbbc00f4e8eb3254ddd349a3ff9 ({username='slave'} at 127.0.99.131:33477)
I20240501 07:59:16.619626 9207 tablet_copy_service.cc:161] P 83f20e84af4c40d09a4804b6b26e697f: Beginning new tablet copy session on tablet c5a876dfb598428ab350593129505504 from peer 3b164cbbc00f4e8eb3254ddd349a3ff9 at {username='slave'} at 127.0.99.131:33477: session id = 3b164cbbc00f4e8eb3254ddd349a3ff9-c5a876dfb598428ab350593129505504
I20240501 07:59:16.620422 9207 tablet_copy_source_session.cc:205] T c5a876dfb598428ab350593129505504 P 83f20e84af4c40d09a4804b6b26e697f: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:16.621263 13132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5a876dfb598428ab350593129505504. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:16.623178 13132 tablet_copy_client.cc:783] T c5a876dfb598428ab350593129505504 P 3b164cbbc00f4e8eb3254ddd349a3ff9: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:16.626569 13132 tablet_copy_client.cc:647] T c5a876dfb598428ab350593129505504 P 3b164cbbc00f4e8eb3254ddd349a3ff9: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:16.627782 13132 tablet_copy_client.cc:523] T c5a876dfb598428ab350593129505504 P 3b164cbbc00f4e8eb3254ddd349a3ff9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:16.628939 13132 tablet_bootstrap.cc:492] T c5a876dfb598428ab350593129505504 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Bootstrap starting.
I20240501 07:59:16.632841 13132 tablet_bootstrap.cc:492] T c5a876dfb598428ab350593129505504 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:16.633139 13132 tablet_bootstrap.cc:492] T c5a876dfb598428ab350593129505504 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Bootstrap complete.
I20240501 07:59:16.633342 13132 ts_tablet_manager.cc:1367] T c5a876dfb598428ab350593129505504 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.005s
I20240501 07:59:16.633772 13132 raft_consensus.cc:348] T c5a876dfb598428ab350593129505504 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: true } }
I20240501 07:59:16.633893 13132 raft_consensus.cc:729] T c5a876dfb598428ab350593129505504 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3b164cbbc00f4e8eb3254ddd349a3ff9, State: Initialized, Role: LEARNER
I20240501 07:59:16.633983 13132 consensus_queue.cc:260] T c5a876dfb598428ab350593129505504 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: true } }
I20240501 07:59:16.634294 13132 ts_tablet_manager.cc:1398] T c5a876dfb598428ab350593129505504 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:59:16.634984 9207 tablet_copy_service.cc:342] P 83f20e84af4c40d09a4804b6b26e697f: Request end of tablet copy session 3b164cbbc00f4e8eb3254ddd349a3ff9-c5a876dfb598428ab350593129505504 received from {username='slave'} at 127.0.99.131:33477
I20240501 07:59:16.635108 9207 tablet_copy_service.cc:433] P 83f20e84af4c40d09a4804b6b26e697f: ending tablet copy session 3b164cbbc00f4e8eb3254ddd349a3ff9-c5a876dfb598428ab350593129505504 on tablet c5a876dfb598428ab350593129505504 with peer 3b164cbbc00f4e8eb3254ddd349a3ff9
I20240501 07:59:16.683275 6165 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55950:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 07:59:16.689182 12850 tablet.cc:1722] T a8e0e297022543f49ea7fb976c5c0606 P a505e7cb133c4392bfda8a4a20874ae9: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 07:59:16.690366 13092 tablet.cc:1722] T a8e0e297022543f49ea7fb976c5c0606 P 83f20e84af4c40d09a4804b6b26e697f: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 07:59:16.690860 13091 tablet.cc:1722] T a8e0e297022543f49ea7fb976c5c0606 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 07:59:16.694630 6165 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 0bc706e85ce74cef97a4827ea4d4aa65: rebalancer_dynamic_table_new_name [id=c06f8c9068414fbd8aa85c90e8970596] alter complete (version 3)
I20240501 07:59:16.742116 12488 raft_consensus.cc:1050] T 12ce6a21d06c4ba889542758be9dc62d P a505e7cb133c4392bfda8a4a20874ae9: attempting to promote NON_VOTER 3b164cbbc00f4e8eb3254ddd349a3ff9 to VOTER
I20240501 07:59:16.742478 12488 consensus_queue.cc:237] T 12ce6a21d06c4ba889542758be9dc62d P a505e7cb133c4392bfda8a4a20874ae9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { replace: true } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: false } }
I20240501 07:59:16.743815 6575 raft_consensus.cc:1261] T 12ce6a21d06c4ba889542758be9dc62d P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 LEARNER]: Refusing update from remote peer a505e7cb133c4392bfda8a4a20874ae9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 07:59:16.743886 8793 raft_consensus.cc:1261] T 12ce6a21d06c4ba889542758be9dc62d P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Refusing update from remote peer a505e7cb133c4392bfda8a4a20874ae9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 07:59:16.744374 11453 consensus_queue.cc:1035] T 12ce6a21d06c4ba889542758be9dc62d P a505e7cb133c4392bfda8a4a20874ae9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 07:59:16.744457 8925 raft_consensus.cc:1261] T 12ce6a21d06c4ba889542758be9dc62d P 13d05ebdca8c4f489c597ce550dd773d [term 1 FOLLOWER]: Refusing update from remote peer a505e7cb133c4392bfda8a4a20874ae9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 07:59:16.744576 13159 consensus_queue.cc:1035] T 12ce6a21d06c4ba889542758be9dc62d P a505e7cb133c4392bfda8a4a20874ae9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 07:59:16.745558 12854 consensus_queue.cc:1035] T 12ce6a21d06c4ba889542758be9dc62d P a505e7cb133c4392bfda8a4a20874ae9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 07:59:16.747104 12119 consensus_queue.cc:1035] T 0b24f67cf1a64debb23ee6618c820d33 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:16.748561 12854 raft_consensus.cc:2932] T 12ce6a21d06c4ba889542758be9dc62d P a505e7cb133c4392bfda8a4a20874ae9 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { replace: true } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: false } } }
I20240501 07:59:16.749509 6577 raft_consensus.cc:2932] T 12ce6a21d06c4ba889542758be9dc62d P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { replace: true } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: false } } }
I20240501 07:59:16.752177 8925 raft_consensus.cc:2932] T 12ce6a21d06c4ba889542758be9dc62d P 13d05ebdca8c4f489c597ce550dd773d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { replace: true } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: false } } }
I20240501 07:59:16.754236 8793 raft_consensus.cc:2932] T 12ce6a21d06c4ba889542758be9dc62d P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { replace: true } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: false } } }
I20240501 07:59:16.754494 6170 catalog_manager.cc:5484] T 12ce6a21d06c4ba889542758be9dc62d P 3b164cbbc00f4e8eb3254ddd349a3ff9 reported cstate change: config changed from index 2 to 3, 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a505e7cb133c4392bfda8a4a20874ae9" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { replace: true } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: false } } }
I20240501 07:59:16.758909 11107 consensus_queue.cc:1035] T 0b24f67cf1a64debb23ee6618c820d33 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:16.760386 9056 consensus_queue.cc:237] T 12ce6a21d06c4ba889542758be9dc62d P a505e7cb133c4392bfda8a4a20874ae9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: false } }
I20240501 07:59:16.761919 8925 raft_consensus.cc:1261] T 12ce6a21d06c4ba889542758be9dc62d P 13d05ebdca8c4f489c597ce550dd773d [term 1 FOLLOWER]: Refusing update from remote peer a505e7cb133c4392bfda8a4a20874ae9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:59:16.762346 6577 raft_consensus.cc:1261] T 12ce6a21d06c4ba889542758be9dc62d P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Refusing update from remote peer a505e7cb133c4392bfda8a4a20874ae9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:59:16.762346 12488 consensus_queue.cc:1035] T 12ce6a21d06c4ba889542758be9dc62d P a505e7cb133c4392bfda8a4a20874ae9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:16.762693 13159 consensus_queue.cc:1035] T 12ce6a21d06c4ba889542758be9dc62d P a505e7cb133c4392bfda8a4a20874ae9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:16.764200 11453 consensus_queue.cc:1035] T c16b7962c18d4e02b850c7c7c678eae4 P a505e7cb133c4392bfda8a4a20874ae9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:16.764442 12488 raft_consensus.cc:2932] T 12ce6a21d06c4ba889542758be9dc62d P a505e7cb133c4392bfda8a4a20874ae9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: false } } }
I20240501 07:59:16.764746 6577 raft_consensus.cc:2932] T 12ce6a21d06c4ba889542758be9dc62d P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: false } } }
I20240501 07:59:16.765401 8925 raft_consensus.cc:2932] T 12ce6a21d06c4ba889542758be9dc62d P 13d05ebdca8c4f489c597ce550dd773d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: false } } }
I20240501 07:59:16.769558 13159 consensus_queue.cc:1035] T c16b7962c18d4e02b850c7c7c678eae4 P a505e7cb133c4392bfda8a4a20874ae9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:16.769642 6164 catalog_manager.cc:5484] T 12ce6a21d06c4ba889542758be9dc62d P 13d05ebdca8c4f489c597ce550dd773d reported cstate change: config changed from index 3 to 4, VOTER 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133) evicted. New cstate: current_term: 1 leader_uuid: "a505e7cb133c4392bfda8a4a20874ae9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: false } } }
I20240501 07:59:16.770217 6158 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 12ce6a21d06c4ba889542758be9dc62d with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 07:59:16.774801 8757 tablet_service.cc:1514] Processing DeleteTablet for tablet 12ce6a21d06c4ba889542758be9dc62d with delete_type TABLET_DATA_TOMBSTONED (TS 74077b61d6f54342bf9736b4a9f2eade not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:43136
I20240501 07:59:16.775157 13180 tablet_replica.cc:330] T 12ce6a21d06c4ba889542758be9dc62d P 74077b61d6f54342bf9736b4a9f2eade: stopping tablet replica
I20240501 07:59:16.775401 13180 raft_consensus.cc:2229] T 12ce6a21d06c4ba889542758be9dc62d P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:16.775576 13180 raft_consensus.cc:2258] T 12ce6a21d06c4ba889542758be9dc62d P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:16.776221 13180 ts_tablet_manager.cc:1875] T 12ce6a21d06c4ba889542758be9dc62d P 74077b61d6f54342bf9736b4a9f2eade: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:59:16.778590 13180 ts_tablet_manager.cc:1888] T 12ce6a21d06c4ba889542758be9dc62d P 74077b61d6f54342bf9736b4a9f2eade: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 07:59:16.778671 13180 log.cc:1196] T 12ce6a21d06c4ba889542758be9dc62d P 74077b61d6f54342bf9736b4a9f2eade: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/12ce6a21d06c4ba889542758be9dc62d
I20240501 07:59:16.779181 6157 catalog_manager.cc:4830] TS 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133:39859): tablet 12ce6a21d06c4ba889542758be9dc62d (table rebalancer_extra_table_10 [id=103f9d5ca0104a7ab2f2f75dead2f70f]) successfully deleted
I20240501 07:59:16.797320 6164 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55950:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 07:59:16.797537 6164 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55950:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 07:59:16.800179 6164 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 0bc706e85ce74cef97a4827ea4d4aa65: Sending DeleteTablet for 3 replicas of tablet a8e0e297022543f49ea7fb976c5c0606
I20240501 07:59:16.800690 9035 tablet_service.cc:1514] Processing DeleteTablet for tablet a8e0e297022543f49ea7fb976c5c0606 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:16 UTC) from {username='slave'} at 127.0.0.1:44236
I20240501 07:59:16.800848 6556 tablet_service.cc:1514] Processing DeleteTablet for tablet a8e0e297022543f49ea7fb976c5c0606 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:16 UTC) from {username='slave'} at 127.0.0.1:53874
I20240501 07:59:16.801021 13197 tablet_replica.cc:330] T a8e0e297022543f49ea7fb976c5c0606 P a505e7cb133c4392bfda8a4a20874ae9: stopping tablet replica
I20240501 07:59:16.801061 9167 tablet_service.cc:1514] Processing DeleteTablet for tablet a8e0e297022543f49ea7fb976c5c0606 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:16 UTC) from {username='slave'} at 127.0.0.1:33594
I20240501 07:59:16.801170 13197 raft_consensus.cc:2229] T a8e0e297022543f49ea7fb976c5c0606 P a505e7cb133c4392bfda8a4a20874ae9 [term 1 LEADER]: Raft consensus shutting down.
I20240501 07:59:16.801347 13199 tablet_replica.cc:330] T a8e0e297022543f49ea7fb976c5c0606 P 83f20e84af4c40d09a4804b6b26e697f: stopping tablet replica
I20240501 07:59:16.801403 13198 tablet_replica.cc:330] T a8e0e297022543f49ea7fb976c5c0606 P 3b164cbbc00f4e8eb3254ddd349a3ff9: stopping tablet replica
I20240501 07:59:16.801446 13197 raft_consensus.cc:2258] T a8e0e297022543f49ea7fb976c5c0606 P a505e7cb133c4392bfda8a4a20874ae9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:16.801537 13199 raft_consensus.cc:2229] T a8e0e297022543f49ea7fb976c5c0606 P 83f20e84af4c40d09a4804b6b26e697f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:16.801633 13198 raft_consensus.cc:2229] T a8e0e297022543f49ea7fb976c5c0606 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:16.801702 13199 raft_consensus.cc:2258] T a8e0e297022543f49ea7fb976c5c0606 P 83f20e84af4c40d09a4804b6b26e697f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:16.801800 13198 raft_consensus.cc:2258] T a8e0e297022543f49ea7fb976c5c0606 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:16.802354 13197 ts_tablet_manager.cc:1875] T a8e0e297022543f49ea7fb976c5c0606 P a505e7cb133c4392bfda8a4a20874ae9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:59:16.802578 13198 ts_tablet_manager.cc:1875] T a8e0e297022543f49ea7fb976c5c0606 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:59:16.802769 13199 ts_tablet_manager.cc:1875] T a8e0e297022543f49ea7fb976c5c0606 P 83f20e84af4c40d09a4804b6b26e697f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:59:16.803452 11107 raft_consensus.cc:1050] T b7d57b7fd5234f50afbc859bc997f3e1 P 3b164cbbc00f4e8eb3254ddd349a3ff9: attempting to promote NON_VOTER 83f20e84af4c40d09a4804b6b26e697f to VOTER
I20240501 07:59:16.803974 11107 consensus_queue.cc:237] T b7d57b7fd5234f50afbc859bc997f3e1 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false } }
I20240501 07:59:16.805495 13198 ts_tablet_manager.cc:1888] T a8e0e297022543f49ea7fb976c5c0606 P 3b164cbbc00f4e8eb3254ddd349a3ff9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:59:16.805613 13198 log.cc:1196] T a8e0e297022543f49ea7fb976c5c0606 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/a8e0e297022543f49ea7fb976c5c0606
I20240501 07:59:16.805671 13199 ts_tablet_manager.cc:1888] T a8e0e297022543f49ea7fb976c5c0606 P 83f20e84af4c40d09a4804b6b26e697f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:59:16.805756 8793 raft_consensus.cc:1261] T b7d57b7fd5234f50afbc859bc997f3e1 P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Refusing update from remote peer 3b164cbbc00f4e8eb3254ddd349a3ff9: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 07:59:16.805799 13197 ts_tablet_manager.cc:1888] T a8e0e297022543f49ea7fb976c5c0606 P a505e7cb133c4392bfda8a4a20874ae9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:59:16.805886 13199 log.cc:1196] T a8e0e297022543f49ea7fb976c5c0606 P 83f20e84af4c40d09a4804b6b26e697f: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/a8e0e297022543f49ea7fb976c5c0606
I20240501 07:59:16.805930 13197 log.cc:1196] T a8e0e297022543f49ea7fb976c5c0606 P a505e7cb133c4392bfda8a4a20874ae9: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/a8e0e297022543f49ea7fb976c5c0606
I20240501 07:59:16.806165 6313 raft_consensus.cc:1261] T b7d57b7fd5234f50afbc859bc997f3e1 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Refusing update from remote peer 3b164cbbc00f4e8eb3254ddd349a3ff9: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 07:59:16.806210 13199 ts_tablet_manager.cc:1909] T a8e0e297022543f49ea7fb976c5c0606 P 83f20e84af4c40d09a4804b6b26e697f: Deleting consensus metadata
I20240501 07:59:16.806351 13197 ts_tablet_manager.cc:1909] T a8e0e297022543f49ea7fb976c5c0606 P a505e7cb133c4392bfda8a4a20874ae9: Deleting consensus metadata
I20240501 07:59:16.806804 10046 consensus_queue.cc:1035] T b7d57b7fd5234f50afbc859bc997f3e1 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 07:59:16.807147 10046 consensus_queue.cc:1035] T b7d57b7fd5234f50afbc859bc997f3e1 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 07:59:16.807228 6158 catalog_manager.cc:4830] TS a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): tablet a8e0e297022543f49ea7fb976c5c0606 (table rebalancer_dynamic_table_new_name [id=c06f8c9068414fbd8aa85c90e8970596]) successfully deleted
I20240501 07:59:16.807431 13198 ts_tablet_manager.cc:1909] T a8e0e297022543f49ea7fb976c5c0606 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Deleting consensus metadata
I20240501 07:59:16.807412 6158 catalog_manager.cc:4830] TS 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): tablet a8e0e297022543f49ea7fb976c5c0606 (table rebalancer_dynamic_table_new_name [id=c06f8c9068414fbd8aa85c90e8970596]) successfully deleted
I20240501 07:59:16.807912 9188 raft_consensus.cc:1261] T b7d57b7fd5234f50afbc859bc997f3e1 P 83f20e84af4c40d09a4804b6b26e697f [term 1 LEARNER]: Refusing update from remote peer 3b164cbbc00f4e8eb3254ddd349a3ff9: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 07:59:16.809427 11119 consensus_queue.cc:1035] T b7d57b7fd5234f50afbc859bc997f3e1 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 07:59:16.809814 6160 catalog_manager.cc:4830] TS 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131:34471): tablet a8e0e297022543f49ea7fb976c5c0606 (table rebalancer_dynamic_table_new_name [id=c06f8c9068414fbd8aa85c90e8970596]) successfully deleted
I20240501 07:59:16.810436 11107 raft_consensus.cc:2932] T b7d57b7fd5234f50afbc859bc997f3e1 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false } } }
I20240501 07:59:16.811551 9188 raft_consensus.cc:2932] T b7d57b7fd5234f50afbc859bc997f3e1 P 83f20e84af4c40d09a4804b6b26e697f [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false } } }
I20240501 07:59:16.812353 8793 raft_consensus.cc:2932] T b7d57b7fd5234f50afbc859bc997f3e1 P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false } } }
I20240501 07:59:16.812383 6313 raft_consensus.cc:2932] T b7d57b7fd5234f50afbc859bc997f3e1 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false } } }
I20240501 07:59:16.815214 6164 catalog_manager.cc:5484] T b7d57b7fd5234f50afbc859bc997f3e1 P 83f20e84af4c40d09a4804b6b26e697f reported cstate change: config changed from index 7 to 8, 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false } } }
I20240501 07:59:16.831561 6165 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55950:
name: "rebalancer_extra_table_24"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 07:59:16.838069 6295 tablet_service.cc:1467] Processing CreateTablet for tablet 0fc1ce00a2024e4a9c19abcc7d181efd (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=635a431f0c2d4f3ca5efbefad804925c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:16.838092 9035 tablet_service.cc:1467] Processing CreateTablet for tablet 0fc1ce00a2024e4a9c19abcc7d181efd (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=635a431f0c2d4f3ca5efbefad804925c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:16.838397 6295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0fc1ce00a2024e4a9c19abcc7d181efd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:16.838407 9035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0fc1ce00a2024e4a9c19abcc7d181efd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:16.839041 8904 tablet_service.cc:1467] Processing CreateTablet for tablet 0fc1ce00a2024e4a9c19abcc7d181efd (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=635a431f0c2d4f3ca5efbefad804925c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:16.839298 8904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0fc1ce00a2024e4a9c19abcc7d181efd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:16.840756 8905 tablet_service.cc:1467] Processing CreateTablet for tablet e5d564b9558946f585ca7a7a3b457a07 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=635a431f0c2d4f3ca5efbefad804925c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:16.841009 8905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5d564b9558946f585ca7a7a3b457a07. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:16.841898 6424 tablet_service.cc:1467] Processing CreateTablet for tablet 13daf8496cd64d7d959ce5ad666b9be8 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=635a431f0c2d4f3ca5efbefad804925c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:16.842192 6424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13daf8496cd64d7d959ce5ad666b9be8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:16.842497 13204 tablet_bootstrap.cc:492] T 0fc1ce00a2024e4a9c19abcc7d181efd P 8b8b3ad3aa8f4053b2167dd550f2086f: Bootstrap starting.
I20240501 07:59:16.843994 9035 tablet_service.cc:1467] Processing CreateTablet for tablet 13daf8496cd64d7d959ce5ad666b9be8 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=635a431f0c2d4f3ca5efbefad804925c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:16.844260 9035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13daf8496cd64d7d959ce5ad666b9be8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:16.844592 13206 tablet_bootstrap.cc:492] T 0fc1ce00a2024e4a9c19abcc7d181efd P 13d05ebdca8c4f489c597ce550dd773d: Bootstrap starting.
I20240501 07:59:16.844679 8757 tablet_service.cc:1467] Processing CreateTablet for tablet e5d564b9558946f585ca7a7a3b457a07 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=635a431f0c2d4f3ca5efbefad804925c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:16.844923 8757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5d564b9558946f585ca7a7a3b457a07. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:16.845515 13206 tablet_bootstrap.cc:654] T 0fc1ce00a2024e4a9c19abcc7d181efd P 13d05ebdca8c4f489c597ce550dd773d: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:16.847363 13209 tablet_bootstrap.cc:492] T e5d564b9558946f585ca7a7a3b457a07 P 74077b61d6f54342bf9736b4a9f2eade: Bootstrap starting.
I20240501 07:59:16.848140 9167 tablet_service.cc:1467] Processing CreateTablet for tablet 13daf8496cd64d7d959ce5ad666b9be8 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=635a431f0c2d4f3ca5efbefad804925c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:16.848306 13209 tablet_bootstrap.cc:654] T e5d564b9558946f585ca7a7a3b457a07 P 74077b61d6f54342bf9736b4a9f2eade: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:16.848444 9167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13daf8496cd64d7d959ce5ad666b9be8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:16.849747 13207 tablet_bootstrap.cc:492] T 13daf8496cd64d7d959ce5ad666b9be8 P 8315d22c56f7439a9e1fc82f153f018c: Bootstrap starting.
I20240501 07:59:16.850363 13209 tablet_bootstrap.cc:492] T e5d564b9558946f585ca7a7a3b457a07 P 74077b61d6f54342bf9736b4a9f2eade: No bootstrap required, opened a new log
I20240501 07:59:16.850438 13209 ts_tablet_manager.cc:1367] T e5d564b9558946f585ca7a7a3b457a07 P 74077b61d6f54342bf9736b4a9f2eade: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240501 07:59:16.850872 13209 raft_consensus.cc:348] T e5d564b9558946f585ca7a7a3b457a07 P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } }
I20240501 07:59:16.850972 13209 raft_consensus.cc:374] T e5d564b9558946f585ca7a7a3b457a07 P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:16.851006 13209 raft_consensus.cc:729] T e5d564b9558946f585ca7a7a3b457a07 P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74077b61d6f54342bf9736b4a9f2eade, State: Initialized, Role: FOLLOWER
I20240501 07:59:16.851119 13209 consensus_queue.cc:260] T e5d564b9558946f585ca7a7a3b457a07 P 74077b61d6f54342bf9736b4a9f2eade [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } }
I20240501 07:59:16.851272 13205 tablet_bootstrap.cc:492] T 0fc1ce00a2024e4a9c19abcc7d181efd P a505e7cb133c4392bfda8a4a20874ae9: Bootstrap starting.
I20240501 07:59:16.851377 13209 ts_tablet_manager.cc:1398] T e5d564b9558946f585ca7a7a3b457a07 P 74077b61d6f54342bf9736b4a9f2eade: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:16.843984 6295 tablet_service.cc:1467] Processing CreateTablet for tablet e5d564b9558946f585ca7a7a3b457a07 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=635a431f0c2d4f3ca5efbefad804925c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:16.851615 6295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5d564b9558946f585ca7a7a3b457a07. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:16.852229 13205 tablet_bootstrap.cc:654] T 0fc1ce00a2024e4a9c19abcc7d181efd P a505e7cb133c4392bfda8a4a20874ae9: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:16.843710 13204 tablet_bootstrap.cc:654] T 0fc1ce00a2024e4a9c19abcc7d181efd P 8b8b3ad3aa8f4053b2167dd550f2086f: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:16.856184 13204 tablet_bootstrap.cc:492] T 0fc1ce00a2024e4a9c19abcc7d181efd P 8b8b3ad3aa8f4053b2167dd550f2086f: No bootstrap required, opened a new log
I20240501 07:59:16.856319 13204 ts_tablet_manager.cc:1367] T 0fc1ce00a2024e4a9c19abcc7d181efd P 8b8b3ad3aa8f4053b2167dd550f2086f: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.000s
I20240501 07:59:16.856978 13204 raft_consensus.cc:348] T 0fc1ce00a2024e4a9c19abcc7d181efd P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } }
I20240501 07:59:16.857170 13204 raft_consensus.cc:374] T 0fc1ce00a2024e4a9c19abcc7d181efd P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:16.857229 13204 raft_consensus.cc:729] T 0fc1ce00a2024e4a9c19abcc7d181efd P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b8b3ad3aa8f4053b2167dd550f2086f, State: Initialized, Role: FOLLOWER
I20240501 07:59:16.857360 13204 consensus_queue.cc:260] T 0fc1ce00a2024e4a9c19abcc7d181efd P 8b8b3ad3aa8f4053b2167dd550f2086f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } }
I20240501 07:59:16.857501 13211 tablet_bootstrap.cc:492] T 13daf8496cd64d7d959ce5ad666b9be8 P 83f20e84af4c40d09a4804b6b26e697f: Bootstrap starting.
I20240501 07:59:16.857755 13204 ts_tablet_manager.cc:1398] T 0fc1ce00a2024e4a9c19abcc7d181efd P 8b8b3ad3aa8f4053b2167dd550f2086f: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 07:59:16.857988 13204 tablet_bootstrap.cc:492] T e5d564b9558946f585ca7a7a3b457a07 P 8b8b3ad3aa8f4053b2167dd550f2086f: Bootstrap starting.
I20240501 07:59:16.858553 13211 tablet_bootstrap.cc:654] T 13daf8496cd64d7d959ce5ad666b9be8 P 83f20e84af4c40d09a4804b6b26e697f: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:16.859207 13204 tablet_bootstrap.cc:654] T e5d564b9558946f585ca7a7a3b457a07 P 8b8b3ad3aa8f4053b2167dd550f2086f: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:16.860587 13204 tablet_bootstrap.cc:492] T e5d564b9558946f585ca7a7a3b457a07 P 8b8b3ad3aa8f4053b2167dd550f2086f: No bootstrap required, opened a new log
I20240501 07:59:16.860662 13204 ts_tablet_manager.cc:1367] T e5d564b9558946f585ca7a7a3b457a07 P 8b8b3ad3aa8f4053b2167dd550f2086f: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240501 07:59:16.860664 13206 tablet_bootstrap.cc:492] T 0fc1ce00a2024e4a9c19abcc7d181efd P 13d05ebdca8c4f489c597ce550dd773d: No bootstrap required, opened a new log
I20240501 07:59:16.860771 13206 ts_tablet_manager.cc:1367] T 0fc1ce00a2024e4a9c19abcc7d181efd P 13d05ebdca8c4f489c597ce550dd773d: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.000s
I20240501 07:59:16.861107 13204 raft_consensus.cc:348] T e5d564b9558946f585ca7a7a3b457a07 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } }
I20240501 07:59:16.861203 13204 raft_consensus.cc:374] T e5d564b9558946f585ca7a7a3b457a07 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:16.861230 13204 raft_consensus.cc:729] T e5d564b9558946f585ca7a7a3b457a07 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b8b3ad3aa8f4053b2167dd550f2086f, State: Initialized, Role: FOLLOWER
I20240501 07:59:16.861260 13206 raft_consensus.cc:348] T 0fc1ce00a2024e4a9c19abcc7d181efd P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } }
I20240501 07:59:16.861363 13206 raft_consensus.cc:374] T 0fc1ce00a2024e4a9c19abcc7d181efd P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:16.861393 13206 raft_consensus.cc:729] T 0fc1ce00a2024e4a9c19abcc7d181efd P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13d05ebdca8c4f489c597ce550dd773d, State: Initialized, Role: FOLLOWER
I20240501 07:59:16.861362 13204 consensus_queue.cc:260] T e5d564b9558946f585ca7a7a3b457a07 P 8b8b3ad3aa8f4053b2167dd550f2086f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } }
I20240501 07:59:16.861513 13206 consensus_queue.cc:260] T 0fc1ce00a2024e4a9c19abcc7d181efd P 13d05ebdca8c4f489c597ce550dd773d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } }
I20240501 07:59:16.861611 13204 ts_tablet_manager.cc:1398] T e5d564b9558946f585ca7a7a3b457a07 P 8b8b3ad3aa8f4053b2167dd550f2086f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:16.861785 13206 ts_tablet_manager.cc:1398] T 0fc1ce00a2024e4a9c19abcc7d181efd P 13d05ebdca8c4f489c597ce550dd773d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:59:16.861922 13206 tablet_bootstrap.cc:492] T e5d564b9558946f585ca7a7a3b457a07 P 13d05ebdca8c4f489c597ce550dd773d: Bootstrap starting.
I20240501 07:59:16.862769 13206 tablet_bootstrap.cc:654] T e5d564b9558946f585ca7a7a3b457a07 P 13d05ebdca8c4f489c597ce550dd773d: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:16.865360 13211 tablet_bootstrap.cc:492] T 13daf8496cd64d7d959ce5ad666b9be8 P 83f20e84af4c40d09a4804b6b26e697f: No bootstrap required, opened a new log
I20240501 07:59:16.865533 13211 ts_tablet_manager.cc:1367] T 13daf8496cd64d7d959ce5ad666b9be8 P 83f20e84af4c40d09a4804b6b26e697f: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240501 07:59:16.865900 13207 tablet_bootstrap.cc:654] T 13daf8496cd64d7d959ce5ad666b9be8 P 8315d22c56f7439a9e1fc82f153f018c: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:16.866070 13211 raft_consensus.cc:348] T 13daf8496cd64d7d959ce5ad666b9be8 P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } }
I20240501 07:59:16.866264 13211 raft_consensus.cc:374] T 13daf8496cd64d7d959ce5ad666b9be8 P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:16.866357 13211 raft_consensus.cc:729] T 13daf8496cd64d7d959ce5ad666b9be8 P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83f20e84af4c40d09a4804b6b26e697f, State: Initialized, Role: FOLLOWER
I20240501 07:59:16.866560 13211 consensus_queue.cc:260] T 13daf8496cd64d7d959ce5ad666b9be8 P 83f20e84af4c40d09a4804b6b26e697f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } }
I20240501 07:59:16.866904 13211 ts_tablet_manager.cc:1398] T 13daf8496cd64d7d959ce5ad666b9be8 P 83f20e84af4c40d09a4804b6b26e697f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:16.866967 13205 tablet_bootstrap.cc:492] T 0fc1ce00a2024e4a9c19abcc7d181efd P a505e7cb133c4392bfda8a4a20874ae9: No bootstrap required, opened a new log
I20240501 07:59:16.867125 13205 ts_tablet_manager.cc:1367] T 0fc1ce00a2024e4a9c19abcc7d181efd P a505e7cb133c4392bfda8a4a20874ae9: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.000s
I20240501 07:59:16.867787 13205 raft_consensus.cc:348] T 0fc1ce00a2024e4a9c19abcc7d181efd P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } }
I20240501 07:59:16.867889 13205 raft_consensus.cc:374] T 0fc1ce00a2024e4a9c19abcc7d181efd P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:16.867923 13205 raft_consensus.cc:729] T 0fc1ce00a2024e4a9c19abcc7d181efd P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a505e7cb133c4392bfda8a4a20874ae9, State: Initialized, Role: FOLLOWER
I20240501 07:59:16.873128 13205 consensus_queue.cc:260] T 0fc1ce00a2024e4a9c19abcc7d181efd P a505e7cb133c4392bfda8a4a20874ae9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } }
I20240501 07:59:16.873642 13205 ts_tablet_manager.cc:1398] T 0fc1ce00a2024e4a9c19abcc7d181efd P a505e7cb133c4392bfda8a4a20874ae9: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20240501 07:59:16.874536 13205 tablet_bootstrap.cc:492] T 13daf8496cd64d7d959ce5ad666b9be8 P a505e7cb133c4392bfda8a4a20874ae9: Bootstrap starting.
I20240501 07:59:16.875118 13206 tablet_bootstrap.cc:492] T e5d564b9558946f585ca7a7a3b457a07 P 13d05ebdca8c4f489c597ce550dd773d: No bootstrap required, opened a new log
I20240501 07:59:16.875227 13206 ts_tablet_manager.cc:1367] T e5d564b9558946f585ca7a7a3b457a07 P 13d05ebdca8c4f489c597ce550dd773d: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.000s
I20240501 07:59:16.875567 13205 tablet_bootstrap.cc:654] T 13daf8496cd64d7d959ce5ad666b9be8 P a505e7cb133c4392bfda8a4a20874ae9: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:16.875761 13206 raft_consensus.cc:348] T e5d564b9558946f585ca7a7a3b457a07 P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } }
I20240501 07:59:16.875878 13206 raft_consensus.cc:374] T e5d564b9558946f585ca7a7a3b457a07 P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:16.875910 13206 raft_consensus.cc:729] T e5d564b9558946f585ca7a7a3b457a07 P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13d05ebdca8c4f489c597ce550dd773d, State: Initialized, Role: FOLLOWER
I20240501 07:59:16.876049 13206 consensus_queue.cc:260] T e5d564b9558946f585ca7a7a3b457a07 P 13d05ebdca8c4f489c597ce550dd773d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } }
I20240501 07:59:16.876298 13206 ts_tablet_manager.cc:1398] T e5d564b9558946f585ca7a7a3b457a07 P 13d05ebdca8c4f489c597ce550dd773d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:16.877496 13205 tablet_bootstrap.cc:492] T 13daf8496cd64d7d959ce5ad666b9be8 P a505e7cb133c4392bfda8a4a20874ae9: No bootstrap required, opened a new log
I20240501 07:59:16.877583 13205 ts_tablet_manager.cc:1367] T 13daf8496cd64d7d959ce5ad666b9be8 P a505e7cb133c4392bfda8a4a20874ae9: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 07:59:16.878125 13205 raft_consensus.cc:348] T 13daf8496cd64d7d959ce5ad666b9be8 P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } }
I20240501 07:59:16.878229 13205 raft_consensus.cc:374] T 13daf8496cd64d7d959ce5ad666b9be8 P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:16.878257 13205 raft_consensus.cc:729] T 13daf8496cd64d7d959ce5ad666b9be8 P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a505e7cb133c4392bfda8a4a20874ae9, State: Initialized, Role: FOLLOWER
I20240501 07:59:16.878711 13207 tablet_bootstrap.cc:492] T 13daf8496cd64d7d959ce5ad666b9be8 P 8315d22c56f7439a9e1fc82f153f018c: No bootstrap required, opened a new log
I20240501 07:59:16.878808 13207 ts_tablet_manager.cc:1367] T 13daf8496cd64d7d959ce5ad666b9be8 P 8315d22c56f7439a9e1fc82f153f018c: Time spent bootstrapping tablet: real 0.029s user 0.002s sys 0.000s
I20240501 07:59:16.879256 13207 raft_consensus.cc:348] T 13daf8496cd64d7d959ce5ad666b9be8 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } }
I20240501 07:59:16.879364 13207 raft_consensus.cc:374] T 13daf8496cd64d7d959ce5ad666b9be8 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:16.879391 13207 raft_consensus.cc:729] T 13daf8496cd64d7d959ce5ad666b9be8 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8315d22c56f7439a9e1fc82f153f018c, State: Initialized, Role: FOLLOWER
I20240501 07:59:16.879534 13207 consensus_queue.cc:260] T 13daf8496cd64d7d959ce5ad666b9be8 P 8315d22c56f7439a9e1fc82f153f018c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } }
I20240501 07:59:16.879787 13207 ts_tablet_manager.cc:1398] T 13daf8496cd64d7d959ce5ad666b9be8 P 8315d22c56f7439a9e1fc82f153f018c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:16.880203 13205 consensus_queue.cc:260] T 13daf8496cd64d7d959ce5ad666b9be8 P a505e7cb133c4392bfda8a4a20874ae9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } }
I20240501 07:59:16.880383 13159 raft_consensus.cc:482] T 13daf8496cd64d7d959ce5ad666b9be8 P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:16.880484 13159 raft_consensus.cc:504] T 13daf8496cd64d7d959ce5ad666b9be8 P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } }
I20240501 07:59:16.880926 13159 leader_election.cc:288] T 13daf8496cd64d7d959ce5ad666b9be8 P a505e7cb133c4392bfda8a4a20874ae9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130:32941), 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149)
I20240501 07:59:16.881598 9188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13daf8496cd64d7d959ce5ad666b9be8" candidate_uuid: "a505e7cb133c4392bfda8a4a20874ae9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83f20e84af4c40d09a4804b6b26e697f" is_pre_election: true
I20240501 07:59:16.881728 9188 raft_consensus.cc:2454] T 13daf8496cd64d7d959ce5ad666b9be8 P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a505e7cb133c4392bfda8a4a20874ae9 in term 0.
I20240501 07:59:16.881966 6445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13daf8496cd64d7d959ce5ad666b9be8" candidate_uuid: "a505e7cb133c4392bfda8a4a20874ae9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8315d22c56f7439a9e1fc82f153f018c" is_pre_election: true
I20240501 07:59:16.882069 6445 raft_consensus.cc:2454] T 13daf8496cd64d7d959ce5ad666b9be8 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a505e7cb133c4392bfda8a4a20874ae9 in term 0.
I20240501 07:59:16.882364 8991 leader_election.cc:302] T 13daf8496cd64d7d959ce5ad666b9be8 P a505e7cb133c4392bfda8a4a20874ae9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8315d22c56f7439a9e1fc82f153f018c, a505e7cb133c4392bfda8a4a20874ae9; no voters:
I20240501 07:59:16.882622 13159 raft_consensus.cc:2781] T 13daf8496cd64d7d959ce5ad666b9be8 P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:16.882684 13159 raft_consensus.cc:482] T 13daf8496cd64d7d959ce5ad666b9be8 P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:16.882720 13159 raft_consensus.cc:3037] T 13daf8496cd64d7d959ce5ad666b9be8 P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:16.882983 13205 ts_tablet_manager.cc:1398] T 13daf8496cd64d7d959ce5ad666b9be8 P a505e7cb133c4392bfda8a4a20874ae9: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240501 07:59:16.883838 13159 raft_consensus.cc:504] T 13daf8496cd64d7d959ce5ad666b9be8 P a505e7cb133c4392bfda8a4a20874ae9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } }
I20240501 07:59:16.884208 13159 leader_election.cc:288] T 13daf8496cd64d7d959ce5ad666b9be8 P a505e7cb133c4392bfda8a4a20874ae9 [CANDIDATE]: Term 1 election: Requested vote from peers 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130:32941), 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149)
I20240501 07:59:16.885032 6445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13daf8496cd64d7d959ce5ad666b9be8" candidate_uuid: "a505e7cb133c4392bfda8a4a20874ae9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8315d22c56f7439a9e1fc82f153f018c"
I20240501 07:59:16.885038 9188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13daf8496cd64d7d959ce5ad666b9be8" candidate_uuid: "a505e7cb133c4392bfda8a4a20874ae9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83f20e84af4c40d09a4804b6b26e697f"
I20240501 07:59:16.885134 6445 raft_consensus.cc:3037] T 13daf8496cd64d7d959ce5ad666b9be8 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:16.885150 9188 raft_consensus.cc:3037] T 13daf8496cd64d7d959ce5ad666b9be8 P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:16.886090 6445 raft_consensus.cc:2454] T 13daf8496cd64d7d959ce5ad666b9be8 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a505e7cb133c4392bfda8a4a20874ae9 in term 1.
I20240501 07:59:16.886176 9188 raft_consensus.cc:2454] T 13daf8496cd64d7d959ce5ad666b9be8 P 83f20e84af4c40d09a4804b6b26e697f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a505e7cb133c4392bfda8a4a20874ae9 in term 1.
I20240501 07:59:16.886426 8991 leader_election.cc:302] T 13daf8496cd64d7d959ce5ad666b9be8 P a505e7cb133c4392bfda8a4a20874ae9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8315d22c56f7439a9e1fc82f153f018c, a505e7cb133c4392bfda8a4a20874ae9; no voters:
I20240501 07:59:16.886896 13159 raft_consensus.cc:2781] T 13daf8496cd64d7d959ce5ad666b9be8 P a505e7cb133c4392bfda8a4a20874ae9 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:16.886976 13159 raft_consensus.cc:686] T 13daf8496cd64d7d959ce5ad666b9be8 P a505e7cb133c4392bfda8a4a20874ae9 [term 1 LEADER]: Becoming Leader. State: Replica: a505e7cb133c4392bfda8a4a20874ae9, State: Running, Role: LEADER
I20240501 07:59:16.887118 13159 consensus_queue.cc:237] T 13daf8496cd64d7d959ce5ad666b9be8 P a505e7cb133c4392bfda8a4a20874ae9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } }
I20240501 07:59:16.891917 6170 catalog_manager.cc:5484] T 13daf8496cd64d7d959ce5ad666b9be8 P a505e7cb133c4392bfda8a4a20874ae9 reported cstate change: term changed from 0 to 1, leader changed from <none> to a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135). New cstate: current_term: 1 leader_uuid: "a505e7cb133c4392bfda8a4a20874ae9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } health_report { overall_health: HEALTHY } } }
I20240501 07:59:16.901599 6170 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55950:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 07:59:16.906765 9035 tablet_service.cc:1467] Processing CreateTablet for tablet a9fa9e5c75894ab5b24195740b0a3f2a (DEFAULT_TABLE table=rebalancer_dynamic_table [id=271831c834f242d196d2660101f49130]), partition=
I20240501 07:59:16.906862 6687 tablet_service.cc:1467] Processing CreateTablet for tablet a9fa9e5c75894ab5b24195740b0a3f2a (DEFAULT_TABLE table=rebalancer_dynamic_table [id=271831c834f242d196d2660101f49130]), partition=
I20240501 07:59:16.906862 9167 tablet_service.cc:1467] Processing CreateTablet for tablet a9fa9e5c75894ab5b24195740b0a3f2a (DEFAULT_TABLE table=rebalancer_dynamic_table [id=271831c834f242d196d2660101f49130]), partition=
I20240501 07:59:16.907055 9035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9fa9e5c75894ab5b24195740b0a3f2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:16.907107 6687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9fa9e5c75894ab5b24195740b0a3f2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:16.907107 9167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9fa9e5c75894ab5b24195740b0a3f2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:16.908660 13205 tablet_bootstrap.cc:492] T a9fa9e5c75894ab5b24195740b0a3f2a P a505e7cb133c4392bfda8a4a20874ae9: Bootstrap starting.
I20240501 07:59:16.909197 13211 tablet_bootstrap.cc:492] T a9fa9e5c75894ab5b24195740b0a3f2a P 83f20e84af4c40d09a4804b6b26e697f: Bootstrap starting.
I20240501 07:59:16.909413 13221 tablet_bootstrap.cc:492] T a9fa9e5c75894ab5b24195740b0a3f2a P 1f9cb72be98941b5a8e504cbe34132b9: Bootstrap starting.
I20240501 07:59:16.910162 13221 tablet_bootstrap.cc:654] T a9fa9e5c75894ab5b24195740b0a3f2a P 1f9cb72be98941b5a8e504cbe34132b9: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:16.910224 13211 tablet_bootstrap.cc:654] T a9fa9e5c75894ab5b24195740b0a3f2a P 83f20e84af4c40d09a4804b6b26e697f: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:16.910531 13205 tablet_bootstrap.cc:654] T a9fa9e5c75894ab5b24195740b0a3f2a P a505e7cb133c4392bfda8a4a20874ae9: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:16.911154 13221 tablet_bootstrap.cc:492] T a9fa9e5c75894ab5b24195740b0a3f2a P 1f9cb72be98941b5a8e504cbe34132b9: No bootstrap required, opened a new log
I20240501 07:59:16.911252 13221 ts_tablet_manager.cc:1367] T a9fa9e5c75894ab5b24195740b0a3f2a P 1f9cb72be98941b5a8e504cbe34132b9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:59:16.911612 13205 tablet_bootstrap.cc:492] T a9fa9e5c75894ab5b24195740b0a3f2a P a505e7cb133c4392bfda8a4a20874ae9: No bootstrap required, opened a new log
I20240501 07:59:16.911700 13205 ts_tablet_manager.cc:1367] T a9fa9e5c75894ab5b24195740b0a3f2a P a505e7cb133c4392bfda8a4a20874ae9: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 07:59:16.912164 13221 raft_consensus.cc:348] T a9fa9e5c75894ab5b24195740b0a3f2a P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:59:16.912312 13221 raft_consensus.cc:374] T a9fa9e5c75894ab5b24195740b0a3f2a P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:16.912372 13221 raft_consensus.cc:729] T a9fa9e5c75894ab5b24195740b0a3f2a P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f9cb72be98941b5a8e504cbe34132b9, State: Initialized, Role: FOLLOWER
I20240501 07:59:16.912322 13205 raft_consensus.cc:348] T a9fa9e5c75894ab5b24195740b0a3f2a P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:59:16.912432 13205 raft_consensus.cc:374] T a9fa9e5c75894ab5b24195740b0a3f2a P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:16.912477 13205 raft_consensus.cc:729] T a9fa9e5c75894ab5b24195740b0a3f2a P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a505e7cb133c4392bfda8a4a20874ae9, State: Initialized, Role: FOLLOWER
I20240501 07:59:16.912529 13221 consensus_queue.cc:260] T a9fa9e5c75894ab5b24195740b0a3f2a P 1f9cb72be98941b5a8e504cbe34132b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:59:16.912667 13205 consensus_queue.cc:260] T a9fa9e5c75894ab5b24195740b0a3f2a P a505e7cb133c4392bfda8a4a20874ae9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:59:16.913064 13221 ts_tablet_manager.cc:1398] T a9fa9e5c75894ab5b24195740b0a3f2a P 1f9cb72be98941b5a8e504cbe34132b9: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:59:16.913074 13205 ts_tablet_manager.cc:1398] T a9fa9e5c75894ab5b24195740b0a3f2a P a505e7cb133c4392bfda8a4a20874ae9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:16.913830 13211 tablet_bootstrap.cc:492] T a9fa9e5c75894ab5b24195740b0a3f2a P 83f20e84af4c40d09a4804b6b26e697f: No bootstrap required, opened a new log
I20240501 07:59:16.913955 13211 ts_tablet_manager.cc:1367] T a9fa9e5c75894ab5b24195740b0a3f2a P 83f20e84af4c40d09a4804b6b26e697f: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 07:59:16.914445 13211 raft_consensus.cc:348] T a9fa9e5c75894ab5b24195740b0a3f2a P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:59:16.914561 13211 raft_consensus.cc:374] T a9fa9e5c75894ab5b24195740b0a3f2a P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:16.914595 13211 raft_consensus.cc:729] T a9fa9e5c75894ab5b24195740b0a3f2a P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83f20e84af4c40d09a4804b6b26e697f, State: Initialized, Role: FOLLOWER
I20240501 07:59:16.914728 13211 consensus_queue.cc:260] T a9fa9e5c75894ab5b24195740b0a3f2a P 83f20e84af4c40d09a4804b6b26e697f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:59:16.915016 13211 ts_tablet_manager.cc:1398] T a9fa9e5c75894ab5b24195740b0a3f2a P 83f20e84af4c40d09a4804b6b26e697f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:16.918658 13225 consensus_queue.cc:1035] T 62936b541ca541faa99eb36bf24c6404 P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:16.922637 11402 consensus_queue.cc:1035] T 62936b541ca541faa99eb36bf24c6404 P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:16.954631 11901 raft_consensus.cc:482] T e5d564b9558946f585ca7a7a3b457a07 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:16.954774 11901 raft_consensus.cc:504] T e5d564b9558946f585ca7a7a3b457a07 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } }
I20240501 07:59:16.955186 11901 leader_election.cc:288] T e5d564b9558946f585ca7a7a3b457a07 P 8b8b3ad3aa8f4053b2167dd550f2086f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134:40141), 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133:39859)
I20240501 07:59:16.955586 8793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5d564b9558946f585ca7a7a3b457a07" candidate_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74077b61d6f54342bf9736b4a9f2eade" is_pre_election: true
I20240501 07:59:16.955588 8925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5d564b9558946f585ca7a7a3b457a07" candidate_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13d05ebdca8c4f489c597ce550dd773d" is_pre_election: true
I20240501 07:59:16.955724 8925 raft_consensus.cc:2454] T e5d564b9558946f585ca7a7a3b457a07 P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b8b3ad3aa8f4053b2167dd550f2086f in term 0.
I20240501 07:59:16.955724 8793 raft_consensus.cc:2454] T e5d564b9558946f585ca7a7a3b457a07 P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b8b3ad3aa8f4053b2167dd550f2086f in term 0.
I20240501 07:59:16.956002 6248 leader_election.cc:302] T e5d564b9558946f585ca7a7a3b457a07 P 8b8b3ad3aa8f4053b2167dd550f2086f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74077b61d6f54342bf9736b4a9f2eade, 8b8b3ad3aa8f4053b2167dd550f2086f; no voters:
I20240501 07:59:16.956168 11901 raft_consensus.cc:2781] T e5d564b9558946f585ca7a7a3b457a07 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:16.956238 11901 raft_consensus.cc:482] T e5d564b9558946f585ca7a7a3b457a07 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:16.956276 11901 raft_consensus.cc:3037] T e5d564b9558946f585ca7a7a3b457a07 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:16.957142 11901 raft_consensus.cc:504] T e5d564b9558946f585ca7a7a3b457a07 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } }
I20240501 07:59:16.957471 11901 leader_election.cc:288] T e5d564b9558946f585ca7a7a3b457a07 P 8b8b3ad3aa8f4053b2167dd550f2086f [CANDIDATE]: Term 1 election: Requested vote from peers 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134:40141), 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133:39859)
I20240501 07:59:16.957867 8793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5d564b9558946f585ca7a7a3b457a07" candidate_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74077b61d6f54342bf9736b4a9f2eade"
I20240501 07:59:16.957885 8925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5d564b9558946f585ca7a7a3b457a07" candidate_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13d05ebdca8c4f489c597ce550dd773d"
I20240501 07:59:16.957984 8793 raft_consensus.cc:3037] T e5d564b9558946f585ca7a7a3b457a07 P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:16.958012 8925 raft_consensus.cc:3037] T e5d564b9558946f585ca7a7a3b457a07 P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:16.958949 8925 raft_consensus.cc:2454] T e5d564b9558946f585ca7a7a3b457a07 P 13d05ebdca8c4f489c597ce550dd773d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b8b3ad3aa8f4053b2167dd550f2086f in term 1.
I20240501 07:59:16.958941 8793 raft_consensus.cc:2454] T e5d564b9558946f585ca7a7a3b457a07 P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b8b3ad3aa8f4053b2167dd550f2086f in term 1.
I20240501 07:59:16.959272 6250 leader_election.cc:302] T e5d564b9558946f585ca7a7a3b457a07 P 8b8b3ad3aa8f4053b2167dd550f2086f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13d05ebdca8c4f489c597ce550dd773d, 8b8b3ad3aa8f4053b2167dd550f2086f; no voters:
I20240501 07:59:16.959434 11901 raft_consensus.cc:2781] T e5d564b9558946f585ca7a7a3b457a07 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:16.959553 11901 raft_consensus.cc:686] T e5d564b9558946f585ca7a7a3b457a07 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 LEADER]: Becoming Leader. State: Replica: 8b8b3ad3aa8f4053b2167dd550f2086f, State: Running, Role: LEADER
I20240501 07:59:16.959702 11901 consensus_queue.cc:237] T e5d564b9558946f585ca7a7a3b457a07 P 8b8b3ad3aa8f4053b2167dd550f2086f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } }
I20240501 07:59:16.961935 6166 catalog_manager.cc:5484] T e5d564b9558946f585ca7a7a3b457a07 P 8b8b3ad3aa8f4053b2167dd550f2086f reported cstate change: term changed from 0 to 1, leader changed from <none> to 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129). New cstate: current_term: 1 leader_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } health_report { overall_health: UNKNOWN } } }
I20240501 07:59:16.996943 12649 raft_consensus.cc:482] T a9fa9e5c75894ab5b24195740b0a3f2a P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:16.997129 12649 raft_consensus.cc:504] T a9fa9e5c75894ab5b24195740b0a3f2a P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:59:16.997725 12649 leader_election.cc:288] T a9fa9e5c75894ab5b24195740b0a3f2a P 83f20e84af4c40d09a4804b6b26e697f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607), 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132:36773)
I20240501 07:59:16.998230 6707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9fa9e5c75894ab5b24195740b0a3f2a" candidate_uuid: "83f20e84af4c40d09a4804b6b26e697f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f9cb72be98941b5a8e504cbe34132b9" is_pre_election: true
I20240501 07:59:16.998225 9056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9fa9e5c75894ab5b24195740b0a3f2a" candidate_uuid: "83f20e84af4c40d09a4804b6b26e697f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a505e7cb133c4392bfda8a4a20874ae9" is_pre_election: true
I20240501 07:59:16.998380 9056 raft_consensus.cc:2454] T a9fa9e5c75894ab5b24195740b0a3f2a P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83f20e84af4c40d09a4804b6b26e697f in term 0.
I20240501 07:59:16.998380 6707 raft_consensus.cc:2454] T a9fa9e5c75894ab5b24195740b0a3f2a P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83f20e84af4c40d09a4804b6b26e697f in term 0.
I20240501 07:59:16.998715 9123 leader_election.cc:302] T a9fa9e5c75894ab5b24195740b0a3f2a P 83f20e84af4c40d09a4804b6b26e697f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83f20e84af4c40d09a4804b6b26e697f, a505e7cb133c4392bfda8a4a20874ae9; no voters:
I20240501 07:59:16.999004 12649 raft_consensus.cc:2781] T a9fa9e5c75894ab5b24195740b0a3f2a P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:16.999066 12649 raft_consensus.cc:482] T a9fa9e5c75894ab5b24195740b0a3f2a P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:16.999118 12649 raft_consensus.cc:3037] T a9fa9e5c75894ab5b24195740b0a3f2a P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:17.000128 12649 raft_consensus.cc:504] T a9fa9e5c75894ab5b24195740b0a3f2a P 83f20e84af4c40d09a4804b6b26e697f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:59:17.000564 12649 leader_election.cc:288] T a9fa9e5c75894ab5b24195740b0a3f2a P 83f20e84af4c40d09a4804b6b26e697f [CANDIDATE]: Term 1 election: Requested vote from peers a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607), 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132:36773)
I20240501 07:59:17.001138 6707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9fa9e5c75894ab5b24195740b0a3f2a" candidate_uuid: "83f20e84af4c40d09a4804b6b26e697f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f9cb72be98941b5a8e504cbe34132b9"
I20240501 07:59:17.001276 6707 raft_consensus.cc:3037] T a9fa9e5c75894ab5b24195740b0a3f2a P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:17.002293 6707 raft_consensus.cc:2454] T a9fa9e5c75894ab5b24195740b0a3f2a P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83f20e84af4c40d09a4804b6b26e697f in term 1.
I20240501 07:59:17.002615 9124 leader_election.cc:302] T a9fa9e5c75894ab5b24195740b0a3f2a P 83f20e84af4c40d09a4804b6b26e697f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f9cb72be98941b5a8e504cbe34132b9, 83f20e84af4c40d09a4804b6b26e697f; no voters:
I20240501 07:59:17.002787 12649 raft_consensus.cc:2781] T a9fa9e5c75894ab5b24195740b0a3f2a P 83f20e84af4c40d09a4804b6b26e697f [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:17.002955 12649 raft_consensus.cc:686] T a9fa9e5c75894ab5b24195740b0a3f2a P 83f20e84af4c40d09a4804b6b26e697f [term 1 LEADER]: Becoming Leader. State: Replica: 83f20e84af4c40d09a4804b6b26e697f, State: Running, Role: LEADER
I20240501 07:59:17.002979 9056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9fa9e5c75894ab5b24195740b0a3f2a" candidate_uuid: "83f20e84af4c40d09a4804b6b26e697f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a505e7cb133c4392bfda8a4a20874ae9"
I20240501 07:59:17.003099 9056 raft_consensus.cc:3037] T a9fa9e5c75894ab5b24195740b0a3f2a P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:17.003122 12649 consensus_queue.cc:237] T a9fa9e5c75894ab5b24195740b0a3f2a P 83f20e84af4c40d09a4804b6b26e697f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } }
I20240501 07:59:17.004173 9056 raft_consensus.cc:2454] T a9fa9e5c75894ab5b24195740b0a3f2a P a505e7cb133c4392bfda8a4a20874ae9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83f20e84af4c40d09a4804b6b26e697f in term 1.
I20240501 07:59:17.005663 6166 catalog_manager.cc:5484] T a9fa9e5c75894ab5b24195740b0a3f2a P 83f20e84af4c40d09a4804b6b26e697f reported cstate change: term changed from 0 to 1, leader changed from <none> to 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136). New cstate: current_term: 1 leader_uuid: "83f20e84af4c40d09a4804b6b26e697f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } health_report { overall_health: UNKNOWN } } }
I20240501 07:59:17.011664 13126 ts_tablet_manager.cc:902] T 5ee738571af740c1b494d4f17413d1d8 P 83f20e84af4c40d09a4804b6b26e697f: Initiating tablet copy from peer 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129:46875)
I20240501 07:59:17.012214 13126 tablet_copy_client.cc:308] T 5ee738571af740c1b494d4f17413d1d8 P 83f20e84af4c40d09a4804b6b26e697f: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:46875
I20240501 07:59:17.012953 6335 tablet_copy_service.cc:140] P 8b8b3ad3aa8f4053b2167dd550f2086f: Received BeginTabletCopySession request for tablet 5ee738571af740c1b494d4f17413d1d8 from peer 83f20e84af4c40d09a4804b6b26e697f ({username='slave'} at 127.0.99.136:55561)
I20240501 07:59:17.013094 6335 tablet_copy_service.cc:161] P 8b8b3ad3aa8f4053b2167dd550f2086f: Beginning new tablet copy session on tablet 5ee738571af740c1b494d4f17413d1d8 from peer 83f20e84af4c40d09a4804b6b26e697f at {username='slave'} at 127.0.99.136:55561: session id = 83f20e84af4c40d09a4804b6b26e697f-5ee738571af740c1b494d4f17413d1d8
I20240501 07:59:17.014101 6335 tablet_copy_source_session.cc:205] T 5ee738571af740c1b494d4f17413d1d8 P 8b8b3ad3aa8f4053b2167dd550f2086f: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:17.015641 13126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ee738571af740c1b494d4f17413d1d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:17.019313 13126 tablet_copy_client.cc:783] T 5ee738571af740c1b494d4f17413d1d8 P 83f20e84af4c40d09a4804b6b26e697f: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:17.019475 13126 tablet_copy_client.cc:647] T 5ee738571af740c1b494d4f17413d1d8 P 83f20e84af4c40d09a4804b6b26e697f: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:17.021135 13126 tablet_copy_client.cc:523] T 5ee738571af740c1b494d4f17413d1d8 P 83f20e84af4c40d09a4804b6b26e697f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:17.021239 9189 raft_consensus.cc:1203] T c3a9036ccfee45498bca9681518d3699 P 83f20e84af4c40d09a4804b6b26e697f [term 2 LEARNER]: Deduplicated request from leader. Original: 2.7->[2.8-2.8] Dedup: 2.8->[]
I20240501 07:59:17.022594 13126 tablet_bootstrap.cc:492] T 5ee738571af740c1b494d4f17413d1d8 P 83f20e84af4c40d09a4804b6b26e697f: Bootstrap starting.
I20240501 07:59:17.041643 6577 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b7d57b7fd5234f50afbc859bc997f3e1"
dest_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:44450
I20240501 07:59:17.041810 6577 raft_consensus.cc:595] T b7d57b7fd5234f50afbc859bc997f3e1 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 LEADER]: Received request to transfer leadership
I20240501 07:59:17.056388 13126 tablet_bootstrap.cc:492] T 5ee738571af740c1b494d4f17413d1d8 P 83f20e84af4c40d09a4804b6b26e697f: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:17.056710 13126 tablet_bootstrap.cc:492] T 5ee738571af740c1b494d4f17413d1d8 P 83f20e84af4c40d09a4804b6b26e697f: Bootstrap complete.
I20240501 07:59:17.056905 13126 ts_tablet_manager.cc:1367] T 5ee738571af740c1b494d4f17413d1d8 P 83f20e84af4c40d09a4804b6b26e697f: Time spent bootstrapping tablet: real 0.034s user 0.020s sys 0.000s
I20240501 07:59:17.058138 13126 raft_consensus.cc:348] T 5ee738571af740c1b494d4f17413d1d8 P 83f20e84af4c40d09a4804b6b26e697f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: true } }
I20240501 07:59:17.058255 13126 raft_consensus.cc:729] T 5ee738571af740c1b494d4f17413d1d8 P 83f20e84af4c40d09a4804b6b26e697f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 83f20e84af4c40d09a4804b6b26e697f, State: Initialized, Role: LEARNER
I20240501 07:59:17.058365 13126 consensus_queue.cc:260] T 5ee738571af740c1b494d4f17413d1d8 P 83f20e84af4c40d09a4804b6b26e697f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: true } }
I20240501 07:59:17.058641 13126 ts_tablet_manager.cc:1398] T 5ee738571af740c1b494d4f17413d1d8 P 83f20e84af4c40d09a4804b6b26e697f: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 07:59:17.059193 6335 tablet_copy_service.cc:342] P 8b8b3ad3aa8f4053b2167dd550f2086f: Request end of tablet copy session 83f20e84af4c40d09a4804b6b26e697f-5ee738571af740c1b494d4f17413d1d8 received from {username='slave'} at 127.0.99.136:55561
I20240501 07:59:17.059289 6335 tablet_copy_service.cc:433] P 8b8b3ad3aa8f4053b2167dd550f2086f: ending tablet copy session 83f20e84af4c40d09a4804b6b26e697f-5ee738571af740c1b494d4f17413d1d8 on tablet 5ee738571af740c1b494d4f17413d1d8 with peer 83f20e84af4c40d09a4804b6b26e697f
I20240501 07:59:17.108875 11901 raft_consensus.cc:482] T 0fc1ce00a2024e4a9c19abcc7d181efd P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:17.109045 11901 raft_consensus.cc:504] T 0fc1ce00a2024e4a9c19abcc7d181efd P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } }
I20240501 07:59:17.109475 11901 leader_election.cc:288] T 0fc1ce00a2024e4a9c19abcc7d181efd P 8b8b3ad3aa8f4053b2167dd550f2086f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607), 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134:40141)
I20240501 07:59:17.109926 8925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fc1ce00a2024e4a9c19abcc7d181efd" candidate_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13d05ebdca8c4f489c597ce550dd773d" is_pre_election: true
I20240501 07:59:17.109928 9056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fc1ce00a2024e4a9c19abcc7d181efd" candidate_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a505e7cb133c4392bfda8a4a20874ae9" is_pre_election: true
I20240501 07:59:17.110083 8925 raft_consensus.cc:2454] T 0fc1ce00a2024e4a9c19abcc7d181efd P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b8b3ad3aa8f4053b2167dd550f2086f in term 0.
I20240501 07:59:17.110093 9056 raft_consensus.cc:2454] T 0fc1ce00a2024e4a9c19abcc7d181efd P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b8b3ad3aa8f4053b2167dd550f2086f in term 0.
I20240501 07:59:17.110391 6250 leader_election.cc:302] T 0fc1ce00a2024e4a9c19abcc7d181efd P 8b8b3ad3aa8f4053b2167dd550f2086f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13d05ebdca8c4f489c597ce550dd773d, 8b8b3ad3aa8f4053b2167dd550f2086f; no voters:
I20240501 07:59:17.110594 11901 raft_consensus.cc:2781] T 0fc1ce00a2024e4a9c19abcc7d181efd P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:17.110653 11901 raft_consensus.cc:482] T 0fc1ce00a2024e4a9c19abcc7d181efd P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:17.110713 11901 raft_consensus.cc:3037] T 0fc1ce00a2024e4a9c19abcc7d181efd P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:17.111812 11901 raft_consensus.cc:504] T 0fc1ce00a2024e4a9c19abcc7d181efd P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } }
I20240501 07:59:17.112203 11901 leader_election.cc:288] T 0fc1ce00a2024e4a9c19abcc7d181efd P 8b8b3ad3aa8f4053b2167dd550f2086f [CANDIDATE]: Term 1 election: Requested vote from peers a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607), 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134:40141)
I20240501 07:59:17.112516 8925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fc1ce00a2024e4a9c19abcc7d181efd" candidate_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13d05ebdca8c4f489c597ce550dd773d"
I20240501 07:59:17.112516 9056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fc1ce00a2024e4a9c19abcc7d181efd" candidate_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a505e7cb133c4392bfda8a4a20874ae9"
I20240501 07:59:17.112643 8925 raft_consensus.cc:3037] T 0fc1ce00a2024e4a9c19abcc7d181efd P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:17.112643 9056 raft_consensus.cc:3037] T 0fc1ce00a2024e4a9c19abcc7d181efd P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:17.113688 8925 raft_consensus.cc:2454] T 0fc1ce00a2024e4a9c19abcc7d181efd P 13d05ebdca8c4f489c597ce550dd773d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b8b3ad3aa8f4053b2167dd550f2086f in term 1.
I20240501 07:59:17.113696 9056 raft_consensus.cc:2454] T 0fc1ce00a2024e4a9c19abcc7d181efd P a505e7cb133c4392bfda8a4a20874ae9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b8b3ad3aa8f4053b2167dd550f2086f in term 1.
I20240501 07:59:17.114009 6250 leader_election.cc:302] T 0fc1ce00a2024e4a9c19abcc7d181efd P 8b8b3ad3aa8f4053b2167dd550f2086f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13d05ebdca8c4f489c597ce550dd773d, 8b8b3ad3aa8f4053b2167dd550f2086f; no voters:
I20240501 07:59:17.114171 11901 raft_consensus.cc:2781] T 0fc1ce00a2024e4a9c19abcc7d181efd P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:17.114257 11901 raft_consensus.cc:686] T 0fc1ce00a2024e4a9c19abcc7d181efd P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 LEADER]: Becoming Leader. State: Replica: 8b8b3ad3aa8f4053b2167dd550f2086f, State: Running, Role: LEADER
I20240501 07:59:17.114378 11901 consensus_queue.cc:237] T 0fc1ce00a2024e4a9c19abcc7d181efd P 8b8b3ad3aa8f4053b2167dd550f2086f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } }
I20240501 07:59:17.116494 6166 catalog_manager.cc:5484] T 0fc1ce00a2024e4a9c19abcc7d181efd P 8b8b3ad3aa8f4053b2167dd550f2086f reported cstate change: term changed from 0 to 1, leader changed from <none> to 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129). New cstate: current_term: 1 leader_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } health_report { overall_health: UNKNOWN } } }
I20240501 07:59:17.133055 6166 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55950:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 07:59:17.138087 9056 raft_consensus.cc:1261] T a9fa9e5c75894ab5b24195740b0a3f2a P a505e7cb133c4392bfda8a4a20874ae9 [term 1 FOLLOWER]: Refusing update from remote peer 83f20e84af4c40d09a4804b6b26e697f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:17.138094 6707 raft_consensus.cc:1261] T a9fa9e5c75894ab5b24195740b0a3f2a P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Refusing update from remote peer 83f20e84af4c40d09a4804b6b26e697f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:17.138478 13235 consensus_queue.cc:1035] T a9fa9e5c75894ab5b24195740b0a3f2a P 83f20e84af4c40d09a4804b6b26e697f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:17.138640 12649 consensus_queue.cc:1035] T a9fa9e5c75894ab5b24195740b0a3f2a P 83f20e84af4c40d09a4804b6b26e697f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:17.141778 6165 catalog_manager.cc:5576] TS a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607) does not have the latest schema for tablet a9fa9e5c75894ab5b24195740b0a3f2a (table rebalancer_dynamic_table [id=271831c834f242d196d2660101f49130]). Expected version 1 got 0
I20240501 07:59:17.142124 6170 catalog_manager.cc:5576] TS 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132:36773) does not have the latest schema for tablet a9fa9e5c75894ab5b24195740b0a3f2a (table rebalancer_dynamic_table [id=271831c834f242d196d2660101f49130]). Expected version 1 got 0
I20240501 07:59:17.142282 13092 tablet.cc:1722] T a9fa9e5c75894ab5b24195740b0a3f2a P 83f20e84af4c40d09a4804b6b26e697f: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 07:59:17.142609 13092 tablet.cc:1620] T a9fa9e5c75894ab5b24195740b0a3f2a P 83f20e84af4c40d09a4804b6b26e697f: MemRowSet was empty: no flush needed.
I20240501 07:59:17.144150 13265 tablet.cc:1722] T a9fa9e5c75894ab5b24195740b0a3f2a P 1f9cb72be98941b5a8e504cbe34132b9: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 07:59:17.144450 13265 tablet.cc:1620] T a9fa9e5c75894ab5b24195740b0a3f2a P 1f9cb72be98941b5a8e504cbe34132b9: MemRowSet was empty: no flush needed.
I20240501 07:59:17.144881 6170 catalog_manager.cc:5576] TS 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132:36773) does not have the latest schema for tablet a9fa9e5c75894ab5b24195740b0a3f2a (table rebalancer_dynamic_table [id=271831c834f242d196d2660101f49130]). Expected version 1 got 0
I20240501 07:59:17.145990 6165 catalog_manager.cc:5576] TS 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149) does not have the latest schema for tablet a9fa9e5c75894ab5b24195740b0a3f2a (table rebalancer_dynamic_table [id=271831c834f242d196d2660101f49130]). Expected version 1 got 0
I20240501 07:59:17.147897 12850 tablet.cc:1722] T a9fa9e5c75894ab5b24195740b0a3f2a P a505e7cb133c4392bfda8a4a20874ae9: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 07:59:17.148218 12850 tablet.cc:1620] T a9fa9e5c75894ab5b24195740b0a3f2a P a505e7cb133c4392bfda8a4a20874ae9: MemRowSet was empty: no flush needed.
I20240501 07:59:17.151295 6170 catalog_manager.cc:5576] TS a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607) does not have the latest schema for tablet a9fa9e5c75894ab5b24195740b0a3f2a (table rebalancer_dynamic_table [id=271831c834f242d196d2660101f49130]). Expected version 1 got 0
I20240501 07:59:17.151499 6165 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 0bc706e85ce74cef97a4827ea4d4aa65: rebalancer_dynamic_table [id=271831c834f242d196d2660101f49130] alter complete (version 1)
I20240501 07:59:17.232599 6577 raft_consensus.cc:1203] T c5a876dfb598428ab350593129505504 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 07:59:17.257133 6170 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55950:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 07:59:17.263130 11107 raft_consensus.cc:979] T b7d57b7fd5234f50afbc859bc997f3e1 P 3b164cbbc00f4e8eb3254ddd349a3ff9: : Instructing follower 83f20e84af4c40d09a4804b6b26e697f to start an election
I20240501 07:59:17.263234 11107 raft_consensus.cc:1067] T b7d57b7fd5234f50afbc859bc997f3e1 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 LEADER]: Signalling peer 83f20e84af4c40d09a4804b6b26e697f to start an election
I20240501 07:59:17.263988 9189 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b7d57b7fd5234f50afbc859bc997f3e1"
dest_uuid: "83f20e84af4c40d09a4804b6b26e697f"
from {username='slave'} at 127.0.99.131:33477
I20240501 07:59:17.264145 9189 raft_consensus.cc:482] T b7d57b7fd5234f50afbc859bc997f3e1 P 83f20e84af4c40d09a4804b6b26e697f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 07:59:17.264207 9189 raft_consensus.cc:3037] T b7d57b7fd5234f50afbc859bc997f3e1 P 83f20e84af4c40d09a4804b6b26e697f [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:17.264330 13092 tablet.cc:1722] T a9fa9e5c75894ab5b24195740b0a3f2a P 83f20e84af4c40d09a4804b6b26e697f: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 07:59:17.264647 13092 tablet.cc:1620] T a9fa9e5c75894ab5b24195740b0a3f2a P 83f20e84af4c40d09a4804b6b26e697f: MemRowSet was empty: no flush needed.
I20240501 07:59:17.264859 12850 tablet.cc:1722] T a9fa9e5c75894ab5b24195740b0a3f2a P a505e7cb133c4392bfda8a4a20874ae9: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 07:59:17.265158 12850 tablet.cc:1620] T a9fa9e5c75894ab5b24195740b0a3f2a P a505e7cb133c4392bfda8a4a20874ae9: MemRowSet was empty: no flush needed.
I20240501 07:59:17.265365 9189 raft_consensus.cc:504] T b7d57b7fd5234f50afbc859bc997f3e1 P 83f20e84af4c40d09a4804b6b26e697f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false } }
I20240501 07:59:17.265792 9189 leader_election.cc:288] T b7d57b7fd5234f50afbc859bc997f3e1 P 83f20e84af4c40d09a4804b6b26e697f [CANDIDATE]: Term 2 election: Requested vote from peers 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131:34471), 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129:46875), 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133:39859)
I20240501 07:59:17.267552 6577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7d57b7fd5234f50afbc859bc997f3e1" candidate_uuid: "83f20e84af4c40d09a4804b6b26e697f" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9"
I20240501 07:59:17.267668 6577 raft_consensus.cc:3032] T b7d57b7fd5234f50afbc859bc997f3e1 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 07:59:17.267699 6577 raft_consensus.cc:729] T b7d57b7fd5234f50afbc859bc997f3e1 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3b164cbbc00f4e8eb3254ddd349a3ff9, State: Running, Role: LEADER
I20240501 07:59:17.267836 6577 consensus_queue.cc:260] T b7d57b7fd5234f50afbc859bc997f3e1 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false } }
I20240501 07:59:17.267994 6577 raft_consensus.cc:3037] T b7d57b7fd5234f50afbc859bc997f3e1 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:17.268337 13265 tablet.cc:1722] T a9fa9e5c75894ab5b24195740b0a3f2a P 1f9cb72be98941b5a8e504cbe34132b9: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 07:59:17.268620 13265 tablet.cc:1620] T a9fa9e5c75894ab5b24195740b0a3f2a P 1f9cb72be98941b5a8e504cbe34132b9: MemRowSet was empty: no flush needed.
I20240501 07:59:17.268913 6577 raft_consensus.cc:2454] T b7d57b7fd5234f50afbc859bc997f3e1 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83f20e84af4c40d09a4804b6b26e697f in term 2.
I20240501 07:59:17.270268 8793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7d57b7fd5234f50afbc859bc997f3e1" candidate_uuid: "83f20e84af4c40d09a4804b6b26e697f" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "74077b61d6f54342bf9736b4a9f2eade"
I20240501 07:59:17.270344 6313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7d57b7fd5234f50afbc859bc997f3e1" candidate_uuid: "83f20e84af4c40d09a4804b6b26e697f" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f"
I20240501 07:59:17.270401 8793 raft_consensus.cc:3037] T b7d57b7fd5234f50afbc859bc997f3e1 P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:17.270471 6313 raft_consensus.cc:3037] T b7d57b7fd5234f50afbc859bc997f3e1 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:17.271464 6313 raft_consensus.cc:2454] T b7d57b7fd5234f50afbc859bc997f3e1 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83f20e84af4c40d09a4804b6b26e697f in term 2.
I20240501 07:59:17.271546 8793 raft_consensus.cc:2454] T b7d57b7fd5234f50afbc859bc997f3e1 P 74077b61d6f54342bf9736b4a9f2eade [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83f20e84af4c40d09a4804b6b26e697f in term 2.
I20240501 07:59:17.273761 6170 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 0bc706e85ce74cef97a4827ea4d4aa65: rebalancer_dynamic_table [id=271831c834f242d196d2660101f49130] alter complete (version 2)
I20240501 07:59:17.275379 9122 leader_election.cc:302] T b7d57b7fd5234f50afbc859bc997f3e1 P 83f20e84af4c40d09a4804b6b26e697f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 3b164cbbc00f4e8eb3254ddd349a3ff9, 74077b61d6f54342bf9736b4a9f2eade, 83f20e84af4c40d09a4804b6b26e697f; no voters:
I20240501 07:59:17.275652 12649 raft_consensus.cc:2781] T b7d57b7fd5234f50afbc859bc997f3e1 P 83f20e84af4c40d09a4804b6b26e697f [term 2 FOLLOWER]: Leader election won for term 2
I20240501 07:59:17.275748 12649 raft_consensus.cc:686] T b7d57b7fd5234f50afbc859bc997f3e1 P 83f20e84af4c40d09a4804b6b26e697f [term 2 LEADER]: Becoming Leader. State: Replica: 83f20e84af4c40d09a4804b6b26e697f, State: Running, Role: LEADER
I20240501 07:59:17.275890 12649 consensus_queue.cc:237] T b7d57b7fd5234f50afbc859bc997f3e1 P 83f20e84af4c40d09a4804b6b26e697f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false } }
I20240501 07:59:17.279565 6164 catalog_manager.cc:5484] T b7d57b7fd5234f50afbc859bc997f3e1 P 83f20e84af4c40d09a4804b6b26e697f reported cstate change: term changed from 1 to 2, leader changed from 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131) to 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136). New cstate: current_term: 2 leader_uuid: "83f20e84af4c40d09a4804b6b26e697f" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 07:59:17.337929 11901 consensus_queue.cc:1035] T e5d564b9558946f585ca7a7a3b457a07 P 8b8b3ad3aa8f4053b2167dd550f2086f [LEADER]: Connected to new peer: Peer: permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:17.341475 11901 consensus_queue.cc:1035] T e5d564b9558946f585ca7a7a3b457a07 P 8b8b3ad3aa8f4053b2167dd550f2086f [LEADER]: Connected to new peer: Peer: permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:17.352185 13159 consensus_queue.cc:1035] T 13daf8496cd64d7d959ce5ad666b9be8 P a505e7cb133c4392bfda8a4a20874ae9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:17.355852 11453 consensus_queue.cc:1035] T 13daf8496cd64d7d959ce5ad666b9be8 P a505e7cb133c4392bfda8a4a20874ae9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:17.378247 6170 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55950:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 07:59:17.383738 13092 tablet.cc:1722] T a9fa9e5c75894ab5b24195740b0a3f2a P 83f20e84af4c40d09a4804b6b26e697f: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 07:59:17.384402 13265 tablet.cc:1722] T a9fa9e5c75894ab5b24195740b0a3f2a P 1f9cb72be98941b5a8e504cbe34132b9: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 07:59:17.385871 12850 tablet.cc:1722] T a9fa9e5c75894ab5b24195740b0a3f2a P a505e7cb133c4392bfda8a4a20874ae9: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 07:59:17.389961 6170 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 0bc706e85ce74cef97a4827ea4d4aa65: rebalancer_dynamic_table_new_name [id=271831c834f242d196d2660101f49130] alter complete (version 3)
I20240501 07:59:17.404767 12937 raft_consensus.cc:1050] T c3a9036ccfee45498bca9681518d3699 P 8315d22c56f7439a9e1fc82f153f018c: attempting to promote NON_VOTER 83f20e84af4c40d09a4804b6b26e697f to VOTER
I20240501 07:59:17.405256 12937 consensus_queue.cc:237] T c3a9036ccfee45498bca9681518d3699 P 8315d22c56f7439a9e1fc82f153f018c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false } }
I20240501 07:59:17.406690 9189 raft_consensus.cc:1261] T c3a9036ccfee45498bca9681518d3699 P 83f20e84af4c40d09a4804b6b26e697f [term 2 LEARNER]: Refusing update from remote peer 8315d22c56f7439a9e1fc82f153f018c: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 07:59:17.406795 8793 raft_consensus.cc:1261] T c3a9036ccfee45498bca9681518d3699 P 74077b61d6f54342bf9736b4a9f2eade [term 2 FOLLOWER]: Refusing update from remote peer 8315d22c56f7439a9e1fc82f153f018c: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 07:59:17.406783 6577 raft_consensus.cc:1261] T c3a9036ccfee45498bca9681518d3699 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 2 FOLLOWER]: Refusing update from remote peer 8315d22c56f7439a9e1fc82f153f018c: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 07:59:17.407142 13297 consensus_queue.cc:1035] T c3a9036ccfee45498bca9681518d3699 P 8315d22c56f7439a9e1fc82f153f018c [LEADER]: Connected to new peer: Peer: permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 07:59:17.407450 13296 consensus_queue.cc:1035] T c3a9036ccfee45498bca9681518d3699 P 8315d22c56f7439a9e1fc82f153f018c [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 07:59:17.407696 12937 consensus_queue.cc:1035] T c3a9036ccfee45498bca9681518d3699 P 8315d22c56f7439a9e1fc82f153f018c [LEADER]: Connected to new peer: Peer: permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 07:59:17.409732 13296 raft_consensus.cc:2932] T c3a9036ccfee45498bca9681518d3699 P 8315d22c56f7439a9e1fc82f153f018c [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false } } }
I20240501 07:59:17.410168 6577 raft_consensus.cc:2932] T c3a9036ccfee45498bca9681518d3699 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false } } }
I20240501 07:59:17.410168 8793 raft_consensus.cc:2932] T c3a9036ccfee45498bca9681518d3699 P 74077b61d6f54342bf9736b4a9f2eade [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false } } }
I20240501 07:59:17.411861 9189 raft_consensus.cc:2932] T c3a9036ccfee45498bca9681518d3699 P 83f20e84af4c40d09a4804b6b26e697f [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false } } }
I20240501 07:59:17.413244 6170 catalog_manager.cc:5484] T c3a9036ccfee45498bca9681518d3699 P 8315d22c56f7439a9e1fc82f153f018c reported cstate change: config changed from index 8 to 9, 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "8315d22c56f7439a9e1fc82f153f018c" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 07:59:17.417217 6444 consensus_queue.cc:237] T c3a9036ccfee45498bca9681518d3699 P 8315d22c56f7439a9e1fc82f153f018c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false } }
I20240501 07:59:17.418485 8793 raft_consensus.cc:1261] T c3a9036ccfee45498bca9681518d3699 P 74077b61d6f54342bf9736b4a9f2eade [term 2 FOLLOWER]: Refusing update from remote peer 8315d22c56f7439a9e1fc82f153f018c: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 07:59:17.419075 9189 raft_consensus.cc:1261] T c3a9036ccfee45498bca9681518d3699 P 83f20e84af4c40d09a4804b6b26e697f [term 2 FOLLOWER]: Refusing update from remote peer 8315d22c56f7439a9e1fc82f153f018c: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 07:59:17.419283 13297 consensus_queue.cc:1035] T c3a9036ccfee45498bca9681518d3699 P 8315d22c56f7439a9e1fc82f153f018c [LEADER]: Connected to new peer: Peer: permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 07:59:17.419617 13296 consensus_queue.cc:1035] T c3a9036ccfee45498bca9681518d3699 P 8315d22c56f7439a9e1fc82f153f018c [LEADER]: Connected to new peer: Peer: permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 07:59:17.421528 13296 raft_consensus.cc:2932] T c3a9036ccfee45498bca9681518d3699 P 8315d22c56f7439a9e1fc82f153f018c [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false } } }
I20240501 07:59:17.421926 8794 raft_consensus.cc:2932] T c3a9036ccfee45498bca9681518d3699 P 74077b61d6f54342bf9736b4a9f2eade [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false } } }
I20240501 07:59:17.423653 9189 raft_consensus.cc:2932] T c3a9036ccfee45498bca9681518d3699 P 83f20e84af4c40d09a4804b6b26e697f [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false } } }
I20240501 07:59:17.424011 6159 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c3a9036ccfee45498bca9681518d3699 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 07:59:17.426206 6165 catalog_manager.cc:5484] T c3a9036ccfee45498bca9681518d3699 P 74077b61d6f54342bf9736b4a9f2eade reported cstate change: config changed from index 9 to 10, VOTER 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131) evicted. New cstate: current_term: 2 leader_uuid: "8315d22c56f7439a9e1fc82f153f018c" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false } } }
I20240501 07:59:17.428830 6556 tablet_service.cc:1514] Processing DeleteTablet for tablet c3a9036ccfee45498bca9681518d3699 with delete_type TABLET_DATA_TOMBSTONED (TS 3b164cbbc00f4e8eb3254ddd349a3ff9 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:53874
I20240501 07:59:17.429427 13303 tablet_replica.cc:330] T c3a9036ccfee45498bca9681518d3699 P 3b164cbbc00f4e8eb3254ddd349a3ff9: stopping tablet replica
I20240501 07:59:17.429602 13303 raft_consensus.cc:2229] T c3a9036ccfee45498bca9681518d3699 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:17.429972 13303 raft_consensus.cc:2258] T c3a9036ccfee45498bca9681518d3699 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:17.430909 13303 ts_tablet_manager.cc:1875] T c3a9036ccfee45498bca9681518d3699 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:59:17.433156 13303 ts_tablet_manager.cc:1888] T c3a9036ccfee45498bca9681518d3699 P 3b164cbbc00f4e8eb3254ddd349a3ff9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9
I20240501 07:59:17.433238 13303 log.cc:1196] T c3a9036ccfee45498bca9681518d3699 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/c3a9036ccfee45498bca9681518d3699
I20240501 07:59:17.433766 6160 catalog_manager.cc:4830] TS 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131:34471): tablet c3a9036ccfee45498bca9681518d3699 (table rebalance_test_table_7 [id=8e242b97203b46d0840cb3b6a649e5a7]) successfully deleted
I20240501 07:59:17.458899 9189 raft_consensus.cc:1203] T 5ee738571af740c1b494d4f17413d1d8 P 83f20e84af4c40d09a4804b6b26e697f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240501 07:59:17.493032 6166 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55950:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 07:59:17.493196 6166 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55950:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 07:59:17.495561 6166 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 0bc706e85ce74cef97a4827ea4d4aa65: Sending DeleteTablet for 3 replicas of tablet a9fa9e5c75894ab5b24195740b0a3f2a
I20240501 07:59:17.496167 9035 tablet_service.cc:1514] Processing DeleteTablet for tablet a9fa9e5c75894ab5b24195740b0a3f2a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:17 UTC) from {username='slave'} at 127.0.0.1:44236
I20240501 07:59:17.496371 6687 tablet_service.cc:1514] Processing DeleteTablet for tablet a9fa9e5c75894ab5b24195740b0a3f2a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:17 UTC) from {username='slave'} at 127.0.0.1:38524
I20240501 07:59:17.496459 9167 tablet_service.cc:1514] Processing DeleteTablet for tablet a9fa9e5c75894ab5b24195740b0a3f2a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:17 UTC) from {username='slave'} at 127.0.0.1:33594
I20240501 07:59:17.496701 13314 tablet_replica.cc:330] T a9fa9e5c75894ab5b24195740b0a3f2a P a505e7cb133c4392bfda8a4a20874ae9: stopping tablet replica
I20240501 07:59:17.496853 13314 raft_consensus.cc:2229] T a9fa9e5c75894ab5b24195740b0a3f2a P a505e7cb133c4392bfda8a4a20874ae9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:17.497046 13317 tablet_replica.cc:330] T a9fa9e5c75894ab5b24195740b0a3f2a P 83f20e84af4c40d09a4804b6b26e697f: stopping tablet replica
I20240501 07:59:17.497049 13314 raft_consensus.cc:2258] T a9fa9e5c75894ab5b24195740b0a3f2a P a505e7cb133c4392bfda8a4a20874ae9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:17.497175 13316 tablet_replica.cc:330] T a9fa9e5c75894ab5b24195740b0a3f2a P 1f9cb72be98941b5a8e504cbe34132b9: stopping tablet replica
I20240501 07:59:17.497273 13317 raft_consensus.cc:2229] T a9fa9e5c75894ab5b24195740b0a3f2a P 83f20e84af4c40d09a4804b6b26e697f [term 1 LEADER]: Raft consensus shutting down.
I20240501 07:59:17.497321 13316 raft_consensus.cc:2229] T a9fa9e5c75894ab5b24195740b0a3f2a P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:17.497505 13316 raft_consensus.cc:2258] T a9fa9e5c75894ab5b24195740b0a3f2a P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:17.497520 13317 raft_consensus.cc:2258] T a9fa9e5c75894ab5b24195740b0a3f2a P 83f20e84af4c40d09a4804b6b26e697f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:17.497866 13314 ts_tablet_manager.cc:1875] T a9fa9e5c75894ab5b24195740b0a3f2a P a505e7cb133c4392bfda8a4a20874ae9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:59:17.498286 13316 ts_tablet_manager.cc:1875] T a9fa9e5c75894ab5b24195740b0a3f2a P 1f9cb72be98941b5a8e504cbe34132b9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:59:17.498327 13317 ts_tablet_manager.cc:1875] T a9fa9e5c75894ab5b24195740b0a3f2a P 83f20e84af4c40d09a4804b6b26e697f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:59:17.500430 13314 ts_tablet_manager.cc:1888] T a9fa9e5c75894ab5b24195740b0a3f2a P a505e7cb133c4392bfda8a4a20874ae9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:59:17.500553 13314 log.cc:1196] T a9fa9e5c75894ab5b24195740b0a3f2a P a505e7cb133c4392bfda8a4a20874ae9: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/a9fa9e5c75894ab5b24195740b0a3f2a
I20240501 07:59:17.500922 13314 ts_tablet_manager.cc:1909] T a9fa9e5c75894ab5b24195740b0a3f2a P a505e7cb133c4392bfda8a4a20874ae9: Deleting consensus metadata
I20240501 07:59:17.500945 13317 ts_tablet_manager.cc:1888] T a9fa9e5c75894ab5b24195740b0a3f2a P 83f20e84af4c40d09a4804b6b26e697f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:59:17.501116 13317 log.cc:1196] T a9fa9e5c75894ab5b24195740b0a3f2a P 83f20e84af4c40d09a4804b6b26e697f: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/a9fa9e5c75894ab5b24195740b0a3f2a
I20240501 07:59:17.501176 13316 ts_tablet_manager.cc:1888] T a9fa9e5c75894ab5b24195740b0a3f2a P 1f9cb72be98941b5a8e504cbe34132b9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:59:17.501296 13316 log.cc:1196] T a9fa9e5c75894ab5b24195740b0a3f2a P 1f9cb72be98941b5a8e504cbe34132b9: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/a9fa9e5c75894ab5b24195740b0a3f2a
I20240501 07:59:17.501626 13316 ts_tablet_manager.cc:1909] T a9fa9e5c75894ab5b24195740b0a3f2a P 1f9cb72be98941b5a8e504cbe34132b9: Deleting consensus metadata
I20240501 07:59:17.501720 13317 ts_tablet_manager.cc:1909] T a9fa9e5c75894ab5b24195740b0a3f2a P 83f20e84af4c40d09a4804b6b26e697f: Deleting consensus metadata
I20240501 07:59:17.502247 6158 catalog_manager.cc:4830] TS a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): tablet a9fa9e5c75894ab5b24195740b0a3f2a (table rebalancer_dynamic_table_new_name [id=271831c834f242d196d2660101f49130]) successfully deleted
I20240501 07:59:17.502348 6159 catalog_manager.cc:4830] TS 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132:36773): tablet a9fa9e5c75894ab5b24195740b0a3f2a (table rebalancer_dynamic_table_new_name [id=271831c834f242d196d2660101f49130]) successfully deleted
I20240501 07:59:17.502570 6158 catalog_manager.cc:4830] TS 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): tablet a9fa9e5c75894ab5b24195740b0a3f2a (table rebalancer_dynamic_table_new_name [id=271831c834f242d196d2660101f49130]) successfully deleted
I20240501 07:59:17.597566 6166 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55950:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 07:59:17.603663 9167 tablet_service.cc:1467] Processing CreateTablet for tablet 29322efc657745e8832e1d8ba1a88071 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=a4c8d5c1ec2c458da759b32ac1f778de]), partition=
I20240501 07:59:17.603806 8757 tablet_service.cc:1467] Processing CreateTablet for tablet 29322efc657745e8832e1d8ba1a88071 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=a4c8d5c1ec2c458da759b32ac1f778de]), partition=
I20240501 07:59:17.603911 8905 tablet_service.cc:1467] Processing CreateTablet for tablet 29322efc657745e8832e1d8ba1a88071 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=a4c8d5c1ec2c458da759b32ac1f778de]), partition=
I20240501 07:59:17.603991 9167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29322efc657745e8832e1d8ba1a88071. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:17.604032 8757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29322efc657745e8832e1d8ba1a88071. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:17.604199 8905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29322efc657745e8832e1d8ba1a88071. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:17.606882 13345 tablet_bootstrap.cc:492] T 29322efc657745e8832e1d8ba1a88071 P 83f20e84af4c40d09a4804b6b26e697f: Bootstrap starting.
I20240501 07:59:17.607102 13343 tablet_bootstrap.cc:492] T 29322efc657745e8832e1d8ba1a88071 P 13d05ebdca8c4f489c597ce550dd773d: Bootstrap starting.
I20240501 07:59:17.607869 13345 tablet_bootstrap.cc:654] T 29322efc657745e8832e1d8ba1a88071 P 83f20e84af4c40d09a4804b6b26e697f: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:17.608304 13343 tablet_bootstrap.cc:654] T 29322efc657745e8832e1d8ba1a88071 P 13d05ebdca8c4f489c597ce550dd773d: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:17.608786 13344 tablet_bootstrap.cc:492] T 29322efc657745e8832e1d8ba1a88071 P 74077b61d6f54342bf9736b4a9f2eade: Bootstrap starting.
I20240501 07:59:17.609566 13343 tablet_bootstrap.cc:492] T 29322efc657745e8832e1d8ba1a88071 P 13d05ebdca8c4f489c597ce550dd773d: No bootstrap required, opened a new log
I20240501 07:59:17.609740 13344 tablet_bootstrap.cc:654] T 29322efc657745e8832e1d8ba1a88071 P 74077b61d6f54342bf9736b4a9f2eade: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:17.609886 13343 ts_tablet_manager.cc:1367] T 29322efc657745e8832e1d8ba1a88071 P 13d05ebdca8c4f489c597ce550dd773d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 07:59:17.610375 13343 raft_consensus.cc:348] T 29322efc657745e8832e1d8ba1a88071 P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } }
I20240501 07:59:17.610502 13343 raft_consensus.cc:374] T 29322efc657745e8832e1d8ba1a88071 P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:17.610536 13343 raft_consensus.cc:729] T 29322efc657745e8832e1d8ba1a88071 P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13d05ebdca8c4f489c597ce550dd773d, State: Initialized, Role: FOLLOWER
I20240501 07:59:17.610644 13345 tablet_bootstrap.cc:492] T 29322efc657745e8832e1d8ba1a88071 P 83f20e84af4c40d09a4804b6b26e697f: No bootstrap required, opened a new log
I20240501 07:59:17.610745 13345 ts_tablet_manager.cc:1367] T 29322efc657745e8832e1d8ba1a88071 P 83f20e84af4c40d09a4804b6b26e697f: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 07:59:17.610705 13343 consensus_queue.cc:260] T 29322efc657745e8832e1d8ba1a88071 P 13d05ebdca8c4f489c597ce550dd773d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } }
I20240501 07:59:17.611008 13343 ts_tablet_manager.cc:1398] T 29322efc657745e8832e1d8ba1a88071 P 13d05ebdca8c4f489c597ce550dd773d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:17.611213 13345 raft_consensus.cc:348] T 29322efc657745e8832e1d8ba1a88071 P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } }
I20240501 07:59:17.611310 13345 raft_consensus.cc:374] T 29322efc657745e8832e1d8ba1a88071 P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:17.611366 13345 raft_consensus.cc:729] T 29322efc657745e8832e1d8ba1a88071 P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83f20e84af4c40d09a4804b6b26e697f, State: Initialized, Role: FOLLOWER
I20240501 07:59:17.611505 13345 consensus_queue.cc:260] T 29322efc657745e8832e1d8ba1a88071 P 83f20e84af4c40d09a4804b6b26e697f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } }
I20240501 07:59:17.611804 13345 ts_tablet_manager.cc:1398] T 29322efc657745e8832e1d8ba1a88071 P 83f20e84af4c40d09a4804b6b26e697f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:17.612238 13344 tablet_bootstrap.cc:492] T 29322efc657745e8832e1d8ba1a88071 P 74077b61d6f54342bf9736b4a9f2eade: No bootstrap required, opened a new log
I20240501 07:59:17.612484 13344 ts_tablet_manager.cc:1367] T 29322efc657745e8832e1d8ba1a88071 P 74077b61d6f54342bf9736b4a9f2eade: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 07:59:17.612922 13344 raft_consensus.cc:348] T 29322efc657745e8832e1d8ba1a88071 P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } }
I20240501 07:59:17.613075 13344 raft_consensus.cc:374] T 29322efc657745e8832e1d8ba1a88071 P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:17.613130 13344 raft_consensus.cc:729] T 29322efc657745e8832e1d8ba1a88071 P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74077b61d6f54342bf9736b4a9f2eade, State: Initialized, Role: FOLLOWER
I20240501 07:59:17.613282 13344 consensus_queue.cc:260] T 29322efc657745e8832e1d8ba1a88071 P 74077b61d6f54342bf9736b4a9f2eade [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } }
I20240501 07:59:17.613520 13344 ts_tablet_manager.cc:1398] T 29322efc657745e8832e1d8ba1a88071 P 74077b61d6f54342bf9736b4a9f2eade: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:17.646212 13304 consensus_queue.cc:1035] T 0fc1ce00a2024e4a9c19abcc7d181efd P 8b8b3ad3aa8f4053b2167dd550f2086f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:17.646710 6166 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55950:
name: "rebalancer_extra_table_26"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 07:59:17.651360 13304 consensus_queue.cc:1035] T 0fc1ce00a2024e4a9c19abcc7d181efd P 8b8b3ad3aa8f4053b2167dd550f2086f [LEADER]: Connected to new peer: Peer: permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:17.654148 6687 tablet_service.cc:1467] Processing CreateTablet for tablet fcb3198c1fa5449781dc84756fe02bff (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=5937a35b015648808765f653d0b6805e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:17.654239 6424 tablet_service.cc:1467] Processing CreateTablet for tablet 9461ad0f65e1489ca102f2558194ce8f (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=5937a35b015648808765f653d0b6805e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:17.654486 6687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fcb3198c1fa5449781dc84756fe02bff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:17.654556 6424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9461ad0f65e1489ca102f2558194ce8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:17.654801 8757 tablet_service.cc:1467] Processing CreateTablet for tablet 4d4f47308d174a92bc73f23f2242be9a (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=5937a35b015648808765f653d0b6805e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:17.655054 8757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d4f47308d174a92bc73f23f2242be9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:17.657508 8774 tablet_service.cc:1467] Processing CreateTablet for tablet fcb3198c1fa5449781dc84756fe02bff (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=5937a35b015648808765f653d0b6805e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:17.657688 13353 tablet_bootstrap.cc:492] T 9461ad0f65e1489ca102f2558194ce8f P 8315d22c56f7439a9e1fc82f153f018c: Bootstrap starting.
I20240501 07:59:17.657668 9167 tablet_service.cc:1467] Processing CreateTablet for tablet 4d4f47308d174a92bc73f23f2242be9a (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=5937a35b015648808765f653d0b6805e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:17.657785 8774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fcb3198c1fa5449781dc84756fe02bff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:17.657956 9167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d4f47308d174a92bc73f23f2242be9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:17.658195 13344 tablet_bootstrap.cc:492] T 4d4f47308d174a92bc73f23f2242be9a P 74077b61d6f54342bf9736b4a9f2eade: Bootstrap starting.
I20240501 07:59:17.658708 13353 tablet_bootstrap.cc:654] T 9461ad0f65e1489ca102f2558194ce8f P 8315d22c56f7439a9e1fc82f153f018c: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:17.659061 13344 tablet_bootstrap.cc:654] T 4d4f47308d174a92bc73f23f2242be9a P 74077b61d6f54342bf9736b4a9f2eade: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:17.660077 13345 tablet_bootstrap.cc:492] T 4d4f47308d174a92bc73f23f2242be9a P 83f20e84af4c40d09a4804b6b26e697f: Bootstrap starting.
I20240501 07:59:17.660188 6556 tablet_service.cc:1467] Processing CreateTablet for tablet 9461ad0f65e1489ca102f2558194ce8f (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=5937a35b015648808765f653d0b6805e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:17.660445 13352 tablet_bootstrap.cc:492] T fcb3198c1fa5449781dc84756fe02bff P 1f9cb72be98941b5a8e504cbe34132b9: Bootstrap starting.
I20240501 07:59:17.660478 6556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9461ad0f65e1489ca102f2558194ce8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:17.661226 6311 raft_consensus.cc:1261] T b7d57b7fd5234f50afbc859bc997f3e1 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 2 FOLLOWER]: Refusing update from remote peer 83f20e84af4c40d09a4804b6b26e697f: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 07:59:17.661475 13352 tablet_bootstrap.cc:654] T fcb3198c1fa5449781dc84756fe02bff P 1f9cb72be98941b5a8e504cbe34132b9: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:17.661442 9035 tablet_service.cc:1467] Processing CreateTablet for tablet 4d4f47308d174a92bc73f23f2242be9a (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=5937a35b015648808765f653d0b6805e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:17.661693 9035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d4f47308d174a92bc73f23f2242be9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:17.661652 9034 tablet_service.cc:1467] Processing CreateTablet for tablet fcb3198c1fa5449781dc84756fe02bff (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=5937a35b015648808765f653d0b6805e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:17.661849 9034 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fcb3198c1fa5449781dc84756fe02bff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:17.662221 13266 consensus_queue.cc:1035] T b7d57b7fd5234f50afbc859bc997f3e1 P 83f20e84af4c40d09a4804b6b26e697f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 07:59:17.664528 9036 tablet_service.cc:1467] Processing CreateTablet for tablet 9461ad0f65e1489ca102f2558194ce8f (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=5937a35b015648808765f653d0b6805e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:17.664811 9036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9461ad0f65e1489ca102f2558194ce8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:17.666118 13345 tablet_bootstrap.cc:654] T 4d4f47308d174a92bc73f23f2242be9a P 83f20e84af4c40d09a4804b6b26e697f: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:17.664806 13357 tablet_bootstrap.cc:492] T fcb3198c1fa5449781dc84756fe02bff P a505e7cb133c4392bfda8a4a20874ae9: Bootstrap starting.
I20240501 07:59:17.669186 13359 tablet_bootstrap.cc:492] T 9461ad0f65e1489ca102f2558194ce8f P 3b164cbbc00f4e8eb3254ddd349a3ff9: Bootstrap starting.
I20240501 07:59:17.670308 13359 tablet_bootstrap.cc:654] T 9461ad0f65e1489ca102f2558194ce8f P 3b164cbbc00f4e8eb3254ddd349a3ff9: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:17.670629 13357 tablet_bootstrap.cc:654] T fcb3198c1fa5449781dc84756fe02bff P a505e7cb133c4392bfda8a4a20874ae9: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:17.671797 13353 tablet_bootstrap.cc:492] T 9461ad0f65e1489ca102f2558194ce8f P 8315d22c56f7439a9e1fc82f153f018c: No bootstrap required, opened a new log
I20240501 07:59:17.671902 13353 ts_tablet_manager.cc:1367] T 9461ad0f65e1489ca102f2558194ce8f P 8315d22c56f7439a9e1fc82f153f018c: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.000s
I20240501 07:59:17.671959 13345 tablet_bootstrap.cc:492] T 4d4f47308d174a92bc73f23f2242be9a P 83f20e84af4c40d09a4804b6b26e697f: No bootstrap required, opened a new log
I20240501 07:59:17.672072 13345 ts_tablet_manager.cc:1367] T 4d4f47308d174a92bc73f23f2242be9a P 83f20e84af4c40d09a4804b6b26e697f: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.001s
I20240501 07:59:17.672420 13353 raft_consensus.cc:348] T 9461ad0f65e1489ca102f2558194ce8f P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } }
I20240501 07:59:17.672545 13353 raft_consensus.cc:374] T 9461ad0f65e1489ca102f2558194ce8f P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:17.672497 13345 raft_consensus.cc:348] T 4d4f47308d174a92bc73f23f2242be9a P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } }
I20240501 07:59:17.672576 13353 raft_consensus.cc:729] T 9461ad0f65e1489ca102f2558194ce8f P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8315d22c56f7439a9e1fc82f153f018c, State: Initialized, Role: FOLLOWER
I20240501 07:59:17.672593 13345 raft_consensus.cc:374] T 4d4f47308d174a92bc73f23f2242be9a P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:17.672626 13345 raft_consensus.cc:729] T 4d4f47308d174a92bc73f23f2242be9a P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83f20e84af4c40d09a4804b6b26e697f, State: Initialized, Role: FOLLOWER
I20240501 07:59:17.672724 13353 consensus_queue.cc:260] T 9461ad0f65e1489ca102f2558194ce8f P 8315d22c56f7439a9e1fc82f153f018c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } }
I20240501 07:59:17.672745 13345 consensus_queue.cc:260] T 4d4f47308d174a92bc73f23f2242be9a P 83f20e84af4c40d09a4804b6b26e697f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } }
I20240501 07:59:17.673054 13345 ts_tablet_manager.cc:1398] T 4d4f47308d174a92bc73f23f2242be9a P 83f20e84af4c40d09a4804b6b26e697f: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 07:59:17.673437 13359 tablet_bootstrap.cc:492] T 9461ad0f65e1489ca102f2558194ce8f P 3b164cbbc00f4e8eb3254ddd349a3ff9: No bootstrap required, opened a new log
I20240501 07:59:17.673516 13359 ts_tablet_manager.cc:1367] T 9461ad0f65e1489ca102f2558194ce8f P 3b164cbbc00f4e8eb3254ddd349a3ff9: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 07:59:17.673964 13359 raft_consensus.cc:348] T 9461ad0f65e1489ca102f2558194ce8f P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } }
I20240501 07:59:17.674077 13359 raft_consensus.cc:374] T 9461ad0f65e1489ca102f2558194ce8f P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:17.674149 13359 raft_consensus.cc:729] T 9461ad0f65e1489ca102f2558194ce8f P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b164cbbc00f4e8eb3254ddd349a3ff9, State: Initialized, Role: FOLLOWER
I20240501 07:59:17.674280 13353 ts_tablet_manager.cc:1398] T 9461ad0f65e1489ca102f2558194ce8f P 8315d22c56f7439a9e1fc82f153f018c: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:59:17.674284 13359 consensus_queue.cc:260] T 9461ad0f65e1489ca102f2558194ce8f P 3b164cbbc00f4e8eb3254ddd349a3ff9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } }
I20240501 07:59:17.674559 13359 ts_tablet_manager.cc:1398] T 9461ad0f65e1489ca102f2558194ce8f P 3b164cbbc00f4e8eb3254ddd349a3ff9: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:59:17.675503 13357 tablet_bootstrap.cc:492] T fcb3198c1fa5449781dc84756fe02bff P a505e7cb133c4392bfda8a4a20874ae9: No bootstrap required, opened a new log
I20240501 07:59:17.675597 13357 ts_tablet_manager.cc:1367] T fcb3198c1fa5449781dc84756fe02bff P a505e7cb133c4392bfda8a4a20874ae9: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240501 07:59:17.676121 13357 raft_consensus.cc:348] T fcb3198c1fa5449781dc84756fe02bff P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } }
I20240501 07:59:17.676235 13357 raft_consensus.cc:374] T fcb3198c1fa5449781dc84756fe02bff P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:17.676265 13357 raft_consensus.cc:729] T fcb3198c1fa5449781dc84756fe02bff P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a505e7cb133c4392bfda8a4a20874ae9, State: Initialized, Role: FOLLOWER
I20240501 07:59:17.676388 13357 consensus_queue.cc:260] T fcb3198c1fa5449781dc84756fe02bff P a505e7cb133c4392bfda8a4a20874ae9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } }
I20240501 07:59:17.676654 13357 ts_tablet_manager.cc:1398] T fcb3198c1fa5449781dc84756fe02bff P a505e7cb133c4392bfda8a4a20874ae9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:17.676826 13357 tablet_bootstrap.cc:492] T 4d4f47308d174a92bc73f23f2242be9a P a505e7cb133c4392bfda8a4a20874ae9: Bootstrap starting.
I20240501 07:59:17.679507 13357 tablet_bootstrap.cc:654] T 4d4f47308d174a92bc73f23f2242be9a P a505e7cb133c4392bfda8a4a20874ae9: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:17.680737 13357 tablet_bootstrap.cc:492] T 4d4f47308d174a92bc73f23f2242be9a P a505e7cb133c4392bfda8a4a20874ae9: No bootstrap required, opened a new log
I20240501 07:59:17.680824 13357 ts_tablet_manager.cc:1367] T 4d4f47308d174a92bc73f23f2242be9a P a505e7cb133c4392bfda8a4a20874ae9: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 07:59:17.681332 13357 raft_consensus.cc:348] T 4d4f47308d174a92bc73f23f2242be9a P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } }
I20240501 07:59:17.681438 13357 raft_consensus.cc:374] T 4d4f47308d174a92bc73f23f2242be9a P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:17.681469 13357 raft_consensus.cc:729] T 4d4f47308d174a92bc73f23f2242be9a P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a505e7cb133c4392bfda8a4a20874ae9, State: Initialized, Role: FOLLOWER
I20240501 07:59:17.681594 13357 consensus_queue.cc:260] T 4d4f47308d174a92bc73f23f2242be9a P a505e7cb133c4392bfda8a4a20874ae9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } }
I20240501 07:59:17.681890 13357 ts_tablet_manager.cc:1398] T 4d4f47308d174a92bc73f23f2242be9a P a505e7cb133c4392bfda8a4a20874ae9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:17.682058 13357 tablet_bootstrap.cc:492] T 9461ad0f65e1489ca102f2558194ce8f P a505e7cb133c4392bfda8a4a20874ae9: Bootstrap starting.
I20240501 07:59:17.682973 13357 tablet_bootstrap.cc:654] T 9461ad0f65e1489ca102f2558194ce8f P a505e7cb133c4392bfda8a4a20874ae9: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:17.684124 13357 tablet_bootstrap.cc:492] T 9461ad0f65e1489ca102f2558194ce8f P a505e7cb133c4392bfda8a4a20874ae9: No bootstrap required, opened a new log
I20240501 07:59:17.684211 13357 ts_tablet_manager.cc:1367] T 9461ad0f65e1489ca102f2558194ce8f P a505e7cb133c4392bfda8a4a20874ae9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:59:17.684893 13357 raft_consensus.cc:348] T 9461ad0f65e1489ca102f2558194ce8f P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } }
I20240501 07:59:17.685086 13357 raft_consensus.cc:374] T 9461ad0f65e1489ca102f2558194ce8f P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:17.685165 13357 raft_consensus.cc:729] T 9461ad0f65e1489ca102f2558194ce8f P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a505e7cb133c4392bfda8a4a20874ae9, State: Initialized, Role: FOLLOWER
I20240501 07:59:17.685304 13357 consensus_queue.cc:260] T 9461ad0f65e1489ca102f2558194ce8f P a505e7cb133c4392bfda8a4a20874ae9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } }
I20240501 07:59:17.685590 13357 ts_tablet_manager.cc:1398] T 9461ad0f65e1489ca102f2558194ce8f P a505e7cb133c4392bfda8a4a20874ae9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:17.686414 13344 tablet_bootstrap.cc:492] T 4d4f47308d174a92bc73f23f2242be9a P 74077b61d6f54342bf9736b4a9f2eade: No bootstrap required, opened a new log
I20240501 07:59:17.686511 13344 ts_tablet_manager.cc:1367] T 4d4f47308d174a92bc73f23f2242be9a P 74077b61d6f54342bf9736b4a9f2eade: Time spent bootstrapping tablet: real 0.028s user 0.003s sys 0.000s
I20240501 07:59:17.686582 13352 tablet_bootstrap.cc:492] T fcb3198c1fa5449781dc84756fe02bff P 1f9cb72be98941b5a8e504cbe34132b9: No bootstrap required, opened a new log
I20240501 07:59:17.686674 13352 ts_tablet_manager.cc:1367] T fcb3198c1fa5449781dc84756fe02bff P 1f9cb72be98941b5a8e504cbe34132b9: Time spent bootstrapping tablet: real 0.026s user 0.002s sys 0.000s
I20240501 07:59:17.687016 13344 raft_consensus.cc:348] T 4d4f47308d174a92bc73f23f2242be9a P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } }
I20240501 07:59:17.687127 13344 raft_consensus.cc:374] T 4d4f47308d174a92bc73f23f2242be9a P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:17.687157 13344 raft_consensus.cc:729] T 4d4f47308d174a92bc73f23f2242be9a P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74077b61d6f54342bf9736b4a9f2eade, State: Initialized, Role: FOLLOWER
I20240501 07:59:17.687098 13352 raft_consensus.cc:348] T fcb3198c1fa5449781dc84756fe02bff P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } }
I20240501 07:59:17.687197 13352 raft_consensus.cc:374] T fcb3198c1fa5449781dc84756fe02bff P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:17.687229 13352 raft_consensus.cc:729] T fcb3198c1fa5449781dc84756fe02bff P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f9cb72be98941b5a8e504cbe34132b9, State: Initialized, Role: FOLLOWER
I20240501 07:59:17.687278 13344 consensus_queue.cc:260] T 4d4f47308d174a92bc73f23f2242be9a P 74077b61d6f54342bf9736b4a9f2eade [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } }
I20240501 07:59:17.687345 13352 consensus_queue.cc:260] T fcb3198c1fa5449781dc84756fe02bff P 1f9cb72be98941b5a8e504cbe34132b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } }
I20240501 07:59:17.687566 13344 ts_tablet_manager.cc:1398] T 4d4f47308d174a92bc73f23f2242be9a P 74077b61d6f54342bf9736b4a9f2eade: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:17.687621 13352 ts_tablet_manager.cc:1398] T fcb3198c1fa5449781dc84756fe02bff P 1f9cb72be98941b5a8e504cbe34132b9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:17.687752 13344 tablet_bootstrap.cc:492] T fcb3198c1fa5449781dc84756fe02bff P 74077b61d6f54342bf9736b4a9f2eade: Bootstrap starting.
I20240501 07:59:17.688701 13344 tablet_bootstrap.cc:654] T fcb3198c1fa5449781dc84756fe02bff P 74077b61d6f54342bf9736b4a9f2eade: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:17.693722 13344 tablet_bootstrap.cc:492] T fcb3198c1fa5449781dc84756fe02bff P 74077b61d6f54342bf9736b4a9f2eade: No bootstrap required, opened a new log
I20240501 07:59:17.693876 13344 ts_tablet_manager.cc:1367] T fcb3198c1fa5449781dc84756fe02bff P 74077b61d6f54342bf9736b4a9f2eade: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 07:59:17.694794 13344 raft_consensus.cc:348] T fcb3198c1fa5449781dc84756fe02bff P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } }
I20240501 07:59:17.695001 13344 raft_consensus.cc:374] T fcb3198c1fa5449781dc84756fe02bff P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:17.695097 13344 raft_consensus.cc:729] T fcb3198c1fa5449781dc84756fe02bff P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74077b61d6f54342bf9736b4a9f2eade, State: Initialized, Role: FOLLOWER
I20240501 07:59:17.695461 13344 consensus_queue.cc:260] T fcb3198c1fa5449781dc84756fe02bff P 74077b61d6f54342bf9736b4a9f2eade [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } }
I20240501 07:59:17.696595 13344 ts_tablet_manager.cc:1398] T fcb3198c1fa5449781dc84756fe02bff P 74077b61d6f54342bf9736b4a9f2eade: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240501 07:59:17.739069 13297 raft_consensus.cc:482] T 9461ad0f65e1489ca102f2558194ce8f P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:17.739194 13297 raft_consensus.cc:504] T 9461ad0f65e1489ca102f2558194ce8f P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } }
I20240501 07:59:17.739652 13297 leader_election.cc:288] T 9461ad0f65e1489ca102f2558194ce8f P 8315d22c56f7439a9e1fc82f153f018c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131:34471), a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607)
I20240501 07:59:17.740020 9056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9461ad0f65e1489ca102f2558194ce8f" candidate_uuid: "8315d22c56f7439a9e1fc82f153f018c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a505e7cb133c4392bfda8a4a20874ae9" is_pre_election: true
I20240501 07:59:17.740171 9056 raft_consensus.cc:2454] T 9461ad0f65e1489ca102f2558194ce8f P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8315d22c56f7439a9e1fc82f153f018c in term 0.
I20240501 07:59:17.740159 6577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9461ad0f65e1489ca102f2558194ce8f" candidate_uuid: "8315d22c56f7439a9e1fc82f153f018c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" is_pre_election: true
I20240501 07:59:17.740319 6577 raft_consensus.cc:2454] T 9461ad0f65e1489ca102f2558194ce8f P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8315d22c56f7439a9e1fc82f153f018c in term 0.
I20240501 07:59:17.740433 6380 leader_election.cc:302] T 9461ad0f65e1489ca102f2558194ce8f P 8315d22c56f7439a9e1fc82f153f018c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8315d22c56f7439a9e1fc82f153f018c, a505e7cb133c4392bfda8a4a20874ae9; no voters:
I20240501 07:59:17.740720 13297 raft_consensus.cc:2781] T 9461ad0f65e1489ca102f2558194ce8f P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:17.740772 13297 raft_consensus.cc:482] T 9461ad0f65e1489ca102f2558194ce8f P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:17.740825 13297 raft_consensus.cc:3037] T 9461ad0f65e1489ca102f2558194ce8f P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:17.742080 13297 raft_consensus.cc:504] T 9461ad0f65e1489ca102f2558194ce8f P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } }
I20240501 07:59:17.742452 13297 leader_election.cc:288] T 9461ad0f65e1489ca102f2558194ce8f P 8315d22c56f7439a9e1fc82f153f018c [CANDIDATE]: Term 1 election: Requested vote from peers 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131:34471), a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607)
I20240501 07:59:17.742754 9056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9461ad0f65e1489ca102f2558194ce8f" candidate_uuid: "8315d22c56f7439a9e1fc82f153f018c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a505e7cb133c4392bfda8a4a20874ae9"
I20240501 07:59:17.742784 6577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9461ad0f65e1489ca102f2558194ce8f" candidate_uuid: "8315d22c56f7439a9e1fc82f153f018c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9"
I20240501 07:59:17.742870 9056 raft_consensus.cc:3037] T 9461ad0f65e1489ca102f2558194ce8f P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:17.742914 6577 raft_consensus.cc:3037] T 9461ad0f65e1489ca102f2558194ce8f P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:17.743607 9056 raft_consensus.cc:2454] T 9461ad0f65e1489ca102f2558194ce8f P a505e7cb133c4392bfda8a4a20874ae9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8315d22c56f7439a9e1fc82f153f018c in term 1.
I20240501 07:59:17.743693 6577 raft_consensus.cc:2454] T 9461ad0f65e1489ca102f2558194ce8f P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8315d22c56f7439a9e1fc82f153f018c in term 1.
I20240501 07:59:17.743896 6380 leader_election.cc:302] T 9461ad0f65e1489ca102f2558194ce8f P 8315d22c56f7439a9e1fc82f153f018c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8315d22c56f7439a9e1fc82f153f018c, a505e7cb133c4392bfda8a4a20874ae9; no voters:
I20240501 07:59:17.744076 13297 raft_consensus.cc:2781] T 9461ad0f65e1489ca102f2558194ce8f P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:17.744168 13297 raft_consensus.cc:686] T 9461ad0f65e1489ca102f2558194ce8f P 8315d22c56f7439a9e1fc82f153f018c [term 1 LEADER]: Becoming Leader. State: Replica: 8315d22c56f7439a9e1fc82f153f018c, State: Running, Role: LEADER
I20240501 07:59:17.744314 13297 consensus_queue.cc:237] T 9461ad0f65e1489ca102f2558194ce8f P 8315d22c56f7439a9e1fc82f153f018c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } }
I20240501 07:59:17.746614 6165 catalog_manager.cc:5484] T 9461ad0f65e1489ca102f2558194ce8f P 8315d22c56f7439a9e1fc82f153f018c reported cstate change: term changed from 0 to 1, leader changed from <none> to 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130). New cstate: current_term: 1 leader_uuid: "8315d22c56f7439a9e1fc82f153f018c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } health_report { overall_health: UNKNOWN } } }
I20240501 07:59:17.753959 13040 raft_consensus.cc:482] T 29322efc657745e8832e1d8ba1a88071 P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:17.754070 13040 raft_consensus.cc:504] T 29322efc657745e8832e1d8ba1a88071 P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } }
I20240501 07:59:17.754456 13040 leader_election.cc:288] T 29322efc657745e8832e1d8ba1a88071 P 13d05ebdca8c4f489c597ce550dd773d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149), 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133:39859)
I20240501 07:59:17.754736 9189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29322efc657745e8832e1d8ba1a88071" candidate_uuid: "13d05ebdca8c4f489c597ce550dd773d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83f20e84af4c40d09a4804b6b26e697f" is_pre_election: true
I20240501 07:59:17.754899 9189 raft_consensus.cc:2454] T 29322efc657745e8832e1d8ba1a88071 P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 13d05ebdca8c4f489c597ce550dd773d in term 0.
I20240501 07:59:17.754858 8794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29322efc657745e8832e1d8ba1a88071" candidate_uuid: "13d05ebdca8c4f489c597ce550dd773d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74077b61d6f54342bf9736b4a9f2eade" is_pre_election: true
I20240501 07:59:17.755013 8794 raft_consensus.cc:2454] T 29322efc657745e8832e1d8ba1a88071 P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 13d05ebdca8c4f489c597ce550dd773d in term 0.
I20240501 07:59:17.755263 8859 leader_election.cc:302] T 29322efc657745e8832e1d8ba1a88071 P 13d05ebdca8c4f489c597ce550dd773d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13d05ebdca8c4f489c597ce550dd773d, 83f20e84af4c40d09a4804b6b26e697f; no voters:
I20240501 07:59:17.755944 13040 raft_consensus.cc:2781] T 29322efc657745e8832e1d8ba1a88071 P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:17.756023 13040 raft_consensus.cc:482] T 29322efc657745e8832e1d8ba1a88071 P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:17.756074 13040 raft_consensus.cc:3037] T 29322efc657745e8832e1d8ba1a88071 P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:17.757021 13040 raft_consensus.cc:504] T 29322efc657745e8832e1d8ba1a88071 P 13d05ebdca8c4f489c597ce550dd773d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } }
I20240501 07:59:17.757340 13040 leader_election.cc:288] T 29322efc657745e8832e1d8ba1a88071 P 13d05ebdca8c4f489c597ce550dd773d [CANDIDATE]: Term 1 election: Requested vote from peers 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149), 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133:39859)
I20240501 07:59:17.757639 9189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29322efc657745e8832e1d8ba1a88071" candidate_uuid: "13d05ebdca8c4f489c597ce550dd773d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83f20e84af4c40d09a4804b6b26e697f"
I20240501 07:59:17.757683 8794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29322efc657745e8832e1d8ba1a88071" candidate_uuid: "13d05ebdca8c4f489c597ce550dd773d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74077b61d6f54342bf9736b4a9f2eade"
I20240501 07:59:17.757761 9189 raft_consensus.cc:3037] T 29322efc657745e8832e1d8ba1a88071 P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:17.757824 8794 raft_consensus.cc:3037] T 29322efc657745e8832e1d8ba1a88071 P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:17.758798 8794 raft_consensus.cc:2454] T 29322efc657745e8832e1d8ba1a88071 P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13d05ebdca8c4f489c597ce550dd773d in term 1.
I20240501 07:59:17.758875 9189 raft_consensus.cc:2454] T 29322efc657745e8832e1d8ba1a88071 P 83f20e84af4c40d09a4804b6b26e697f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13d05ebdca8c4f489c597ce550dd773d in term 1.
I20240501 07:59:17.759158 8859 leader_election.cc:302] T 29322efc657745e8832e1d8ba1a88071 P 13d05ebdca8c4f489c597ce550dd773d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13d05ebdca8c4f489c597ce550dd773d, 83f20e84af4c40d09a4804b6b26e697f; no voters:
I20240501 07:59:17.759398 13040 raft_consensus.cc:2781] T 29322efc657745e8832e1d8ba1a88071 P 13d05ebdca8c4f489c597ce550dd773d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:17.759492 13040 raft_consensus.cc:686] T 29322efc657745e8832e1d8ba1a88071 P 13d05ebdca8c4f489c597ce550dd773d [term 1 LEADER]: Becoming Leader. State: Replica: 13d05ebdca8c4f489c597ce550dd773d, State: Running, Role: LEADER
I20240501 07:59:17.759635 13040 consensus_queue.cc:237] T 29322efc657745e8832e1d8ba1a88071 P 13d05ebdca8c4f489c597ce550dd773d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } }
I20240501 07:59:17.761847 6165 catalog_manager.cc:5484] T 29322efc657745e8832e1d8ba1a88071 P 13d05ebdca8c4f489c597ce550dd773d reported cstate change: term changed from 0 to 1, leader changed from <none> to 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134). New cstate: current_term: 1 leader_uuid: "13d05ebdca8c4f489c597ce550dd773d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } health_report { overall_health: UNKNOWN } } }
I20240501 07:59:17.776556 13266 raft_consensus.cc:1050] T c5a876dfb598428ab350593129505504 P 83f20e84af4c40d09a4804b6b26e697f: attempting to promote NON_VOTER 3b164cbbc00f4e8eb3254ddd349a3ff9 to VOTER
I20240501 07:59:17.777022 13266 consensus_queue.cc:237] T c5a876dfb598428ab350593129505504 P 83f20e84af4c40d09a4804b6b26e697f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: false } }
I20240501 07:59:17.778441 6444 raft_consensus.cc:1261] T c5a876dfb598428ab350593129505504 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Refusing update from remote peer 83f20e84af4c40d09a4804b6b26e697f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 07:59:17.778546 8794 raft_consensus.cc:1261] T c5a876dfb598428ab350593129505504 P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Refusing update from remote peer 83f20e84af4c40d09a4804b6b26e697f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 07:59:17.778843 6577 raft_consensus.cc:1261] T c5a876dfb598428ab350593129505504 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 LEARNER]: Refusing update from remote peer 83f20e84af4c40d09a4804b6b26e697f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 07:59:17.779050 13383 consensus_queue.cc:1035] T c5a876dfb598428ab350593129505504 P 83f20e84af4c40d09a4804b6b26e697f [LEADER]: Connected to new peer: Peer: permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 07:59:17.779234 13266 consensus_queue.cc:1035] T c5a876dfb598428ab350593129505504 P 83f20e84af4c40d09a4804b6b26e697f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 07:59:17.779425 13235 consensus_queue.cc:1035] T c5a876dfb598428ab350593129505504 P 83f20e84af4c40d09a4804b6b26e697f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 07:59:17.781387 13383 raft_consensus.cc:2932] T c5a876dfb598428ab350593129505504 P 83f20e84af4c40d09a4804b6b26e697f [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: false } } }
I20240501 07:59:17.782006 6577 raft_consensus.cc:2932] T c5a876dfb598428ab350593129505504 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: false } } }
I20240501 07:59:17.782418 8794 raft_consensus.cc:2932] T c5a876dfb598428ab350593129505504 P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: false } } }
I20240501 07:59:17.785063 6166 catalog_manager.cc:5484] T c5a876dfb598428ab350593129505504 P 83f20e84af4c40d09a4804b6b26e697f reported cstate change: config changed from index 2 to 3, 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "83f20e84af4c40d09a4804b6b26e697f" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 07:59:17.786171 6444 raft_consensus.cc:2932] T c5a876dfb598428ab350593129505504 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { replace: true } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: false } } }
I20240501 07:59:17.789425 9189 consensus_queue.cc:237] T c5a876dfb598428ab350593129505504 P 83f20e84af4c40d09a4804b6b26e697f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: false } }
I20240501 07:59:17.790644 6577 raft_consensus.cc:1261] T c5a876dfb598428ab350593129505504 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Refusing update from remote peer 83f20e84af4c40d09a4804b6b26e697f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:59:17.790688 8794 raft_consensus.cc:1261] T c5a876dfb598428ab350593129505504 P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Refusing update from remote peer 83f20e84af4c40d09a4804b6b26e697f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:59:17.791018 13266 consensus_queue.cc:1035] T c5a876dfb598428ab350593129505504 P 83f20e84af4c40d09a4804b6b26e697f [LEADER]: Connected to new peer: Peer: permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:17.791205 13235 consensus_queue.cc:1035] T c5a876dfb598428ab350593129505504 P 83f20e84af4c40d09a4804b6b26e697f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:17.792665 12649 raft_consensus.cc:2932] T c5a876dfb598428ab350593129505504 P 83f20e84af4c40d09a4804b6b26e697f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: false } } }
I20240501 07:59:17.793084 8794 raft_consensus.cc:2932] T c5a876dfb598428ab350593129505504 P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: false } } }
I20240501 07:59:17.794051 6158 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c5a876dfb598428ab350593129505504 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 07:59:17.795593 6170 catalog_manager.cc:5484] T c5a876dfb598428ab350593129505504 P 83f20e84af4c40d09a4804b6b26e697f reported cstate change: config changed from index 3 to 4, VOTER 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "83f20e84af4c40d09a4804b6b26e697f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 07:59:17.795790 6577 raft_consensus.cc:2932] T c5a876dfb598428ab350593129505504 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: false } } }
I20240501 07:59:17.799227 6424 tablet_service.cc:1514] Processing DeleteTablet for tablet c5a876dfb598428ab350593129505504 with delete_type TABLET_DATA_TOMBSTONED (TS 8315d22c56f7439a9e1fc82f153f018c not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:45034
I20240501 07:59:17.799718 13387 tablet_replica.cc:330] T c5a876dfb598428ab350593129505504 P 8315d22c56f7439a9e1fc82f153f018c: stopping tablet replica
I20240501 07:59:17.799870 13387 raft_consensus.cc:2229] T c5a876dfb598428ab350593129505504 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:17.800045 13387 raft_consensus.cc:2258] T c5a876dfb598428ab350593129505504 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:17.800719 13387 ts_tablet_manager.cc:1875] T c5a876dfb598428ab350593129505504 P 8315d22c56f7439a9e1fc82f153f018c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:59:17.802640 13387 ts_tablet_manager.cc:1888] T c5a876dfb598428ab350593129505504 P 8315d22c56f7439a9e1fc82f153f018c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 07:59:17.802726 13387 log.cc:1196] T c5a876dfb598428ab350593129505504 P 8315d22c56f7439a9e1fc82f153f018c: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/c5a876dfb598428ab350593129505504
I20240501 07:59:17.803292 6159 catalog_manager.cc:4830] TS 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130:32941): tablet c5a876dfb598428ab350593129505504 (table rebalancer_extra_table_2 [id=5e1db4e06c5e470a9d2f8a20dd23c585]) successfully deleted
I20240501 07:59:17.817492 13266 raft_consensus.cc:482] T 4d4f47308d174a92bc73f23f2242be9a P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:17.817637 13266 raft_consensus.cc:504] T 4d4f47308d174a92bc73f23f2242be9a P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } }
I20240501 07:59:17.818080 13266 leader_election.cc:288] T 4d4f47308d174a92bc73f23f2242be9a P 83f20e84af4c40d09a4804b6b26e697f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133:39859), a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607)
I20240501 07:59:17.818511 9056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d4f47308d174a92bc73f23f2242be9a" candidate_uuid: "83f20e84af4c40d09a4804b6b26e697f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a505e7cb133c4392bfda8a4a20874ae9" is_pre_election: true
I20240501 07:59:17.818542 8794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d4f47308d174a92bc73f23f2242be9a" candidate_uuid: "83f20e84af4c40d09a4804b6b26e697f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74077b61d6f54342bf9736b4a9f2eade" is_pre_election: true
I20240501 07:59:17.818652 9056 raft_consensus.cc:2454] T 4d4f47308d174a92bc73f23f2242be9a P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83f20e84af4c40d09a4804b6b26e697f in term 0.
I20240501 07:59:17.818707 8794 raft_consensus.cc:2454] T 4d4f47308d174a92bc73f23f2242be9a P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83f20e84af4c40d09a4804b6b26e697f in term 0.
I20240501 07:59:17.818953 9123 leader_election.cc:302] T 4d4f47308d174a92bc73f23f2242be9a P 83f20e84af4c40d09a4804b6b26e697f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83f20e84af4c40d09a4804b6b26e697f, a505e7cb133c4392bfda8a4a20874ae9; no voters:
I20240501 07:59:17.819110 13266 raft_consensus.cc:2781] T 4d4f47308d174a92bc73f23f2242be9a P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:17.819169 13266 raft_consensus.cc:482] T 4d4f47308d174a92bc73f23f2242be9a P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:17.819196 13266 raft_consensus.cc:3037] T 4d4f47308d174a92bc73f23f2242be9a P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:17.820180 13266 raft_consensus.cc:504] T 4d4f47308d174a92bc73f23f2242be9a P 83f20e84af4c40d09a4804b6b26e697f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } }
I20240501 07:59:17.820544 13266 leader_election.cc:288] T 4d4f47308d174a92bc73f23f2242be9a P 83f20e84af4c40d09a4804b6b26e697f [CANDIDATE]: Term 1 election: Requested vote from peers 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133:39859), a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607)
I20240501 07:59:17.820909 8794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d4f47308d174a92bc73f23f2242be9a" candidate_uuid: "83f20e84af4c40d09a4804b6b26e697f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74077b61d6f54342bf9736b4a9f2eade"
I20240501 07:59:17.820995 9056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d4f47308d174a92bc73f23f2242be9a" candidate_uuid: "83f20e84af4c40d09a4804b6b26e697f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a505e7cb133c4392bfda8a4a20874ae9"
I20240501 07:59:17.821069 8794 raft_consensus.cc:3037] T 4d4f47308d174a92bc73f23f2242be9a P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:17.821113 9056 raft_consensus.cc:3037] T 4d4f47308d174a92bc73f23f2242be9a P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:17.822052 8794 raft_consensus.cc:2454] T 4d4f47308d174a92bc73f23f2242be9a P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83f20e84af4c40d09a4804b6b26e697f in term 1.
I20240501 07:59:17.822055 9056 raft_consensus.cc:2454] T 4d4f47308d174a92bc73f23f2242be9a P a505e7cb133c4392bfda8a4a20874ae9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83f20e84af4c40d09a4804b6b26e697f in term 1.
I20240501 07:59:17.822404 9122 leader_election.cc:302] T 4d4f47308d174a92bc73f23f2242be9a P 83f20e84af4c40d09a4804b6b26e697f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74077b61d6f54342bf9736b4a9f2eade, 83f20e84af4c40d09a4804b6b26e697f; no voters:
I20240501 07:59:17.822727 13266 raft_consensus.cc:2781] T 4d4f47308d174a92bc73f23f2242be9a P 83f20e84af4c40d09a4804b6b26e697f [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:17.823077 13266 raft_consensus.cc:686] T 4d4f47308d174a92bc73f23f2242be9a P 83f20e84af4c40d09a4804b6b26e697f [term 1 LEADER]: Becoming Leader. State: Replica: 83f20e84af4c40d09a4804b6b26e697f, State: Running, Role: LEADER
I20240501 07:59:17.823225 13266 consensus_queue.cc:237] T 4d4f47308d174a92bc73f23f2242be9a P 83f20e84af4c40d09a4804b6b26e697f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } }
I20240501 07:59:17.824784 8794 raft_consensus.cc:1261] T b7d57b7fd5234f50afbc859bc997f3e1 P 74077b61d6f54342bf9736b4a9f2eade [term 2 FOLLOWER]: Refusing update from remote peer 83f20e84af4c40d09a4804b6b26e697f: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 07:59:17.825204 13235 consensus_queue.cc:1035] T b7d57b7fd5234f50afbc859bc997f3e1 P 83f20e84af4c40d09a4804b6b26e697f [LEADER]: Connected to new peer: Peer: permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 07:59:17.825536 6165 catalog_manager.cc:5484] T 4d4f47308d174a92bc73f23f2242be9a P 83f20e84af4c40d09a4804b6b26e697f reported cstate change: term changed from 0 to 1, leader changed from <none> to 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136). New cstate: current_term: 1 leader_uuid: "83f20e84af4c40d09a4804b6b26e697f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } health_report { overall_health: UNKNOWN } } }
I20240501 07:59:17.833561 6577 raft_consensus.cc:1261] T b7d57b7fd5234f50afbc859bc997f3e1 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 2 FOLLOWER]: Refusing update from remote peer 83f20e84af4c40d09a4804b6b26e697f: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 07:59:17.833971 13266 consensus_queue.cc:1035] T b7d57b7fd5234f50afbc859bc997f3e1 P 83f20e84af4c40d09a4804b6b26e697f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 07:59:17.845177 9189 consensus_queue.cc:237] T b7d57b7fd5234f50afbc859bc997f3e1 P 83f20e84af4c40d09a4804b6b26e697f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false } }
I20240501 07:59:17.846853 6311 raft_consensus.cc:1261] T b7d57b7fd5234f50afbc859bc997f3e1 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 2 FOLLOWER]: Refusing update from remote peer 83f20e84af4c40d09a4804b6b26e697f: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 07:59:17.847164 8794 raft_consensus.cc:1261] T b7d57b7fd5234f50afbc859bc997f3e1 P 74077b61d6f54342bf9736b4a9f2eade [term 2 FOLLOWER]: Refusing update from remote peer 83f20e84af4c40d09a4804b6b26e697f: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 07:59:17.847206 13383 consensus_queue.cc:1035] T b7d57b7fd5234f50afbc859bc997f3e1 P 83f20e84af4c40d09a4804b6b26e697f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 07:59:17.848485 13383 consensus_queue.cc:1035] T b7d57b7fd5234f50afbc859bc997f3e1 P 83f20e84af4c40d09a4804b6b26e697f [LEADER]: Connected to new peer: Peer: permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 07:59:17.850369 13266 raft_consensus.cc:2932] T b7d57b7fd5234f50afbc859bc997f3e1 P 83f20e84af4c40d09a4804b6b26e697f [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false } } }
I20240501 07:59:17.852041 6158 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b7d57b7fd5234f50afbc859bc997f3e1 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 07:59:17.852934 6311 raft_consensus.cc:2932] T b7d57b7fd5234f50afbc859bc997f3e1 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false } } }
I20240501 07:59:17.854722 8794 raft_consensus.cc:2932] T b7d57b7fd5234f50afbc859bc997f3e1 P 74077b61d6f54342bf9736b4a9f2eade [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false } } }
I20240501 07:59:17.857215 6165 catalog_manager.cc:5484] T b7d57b7fd5234f50afbc859bc997f3e1 P 8b8b3ad3aa8f4053b2167dd550f2086f reported cstate change: config changed from index 8 to 10, VOTER 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131) evicted. New cstate: current_term: 2 leader_uuid: "83f20e84af4c40d09a4804b6b26e697f" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false } } }
I20240501 07:59:17.859916 6556 tablet_service.cc:1514] Processing DeleteTablet for tablet b7d57b7fd5234f50afbc859bc997f3e1 with delete_type TABLET_DATA_TOMBSTONED (TS 3b164cbbc00f4e8eb3254ddd349a3ff9 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:53874
I20240501 07:59:17.860172 13303 tablet_replica.cc:330] T b7d57b7fd5234f50afbc859bc997f3e1 P 3b164cbbc00f4e8eb3254ddd349a3ff9: stopping tablet replica
I20240501 07:59:17.860325 13303 raft_consensus.cc:2229] T b7d57b7fd5234f50afbc859bc997f3e1 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:17.860438 13303 raft_consensus.cc:2258] T b7d57b7fd5234f50afbc859bc997f3e1 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:17.861315 13303 ts_tablet_manager.cc:1875] T b7d57b7fd5234f50afbc859bc997f3e1 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:59:17.863638 13303 ts_tablet_manager.cc:1888] T b7d57b7fd5234f50afbc859bc997f3e1 P 3b164cbbc00f4e8eb3254ddd349a3ff9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9
I20240501 07:59:17.863729 13303 log.cc:1196] T b7d57b7fd5234f50afbc859bc997f3e1 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/b7d57b7fd5234f50afbc859bc997f3e1
I20240501 07:59:17.864487 6160 catalog_manager.cc:4830] TS 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131:34471): tablet b7d57b7fd5234f50afbc859bc997f3e1 (table rebalance_test_table_11 [id=3e062fd8d58941efbd3d190c28bb0059]) successfully deleted
I20240501 07:59:17.872308 6170 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55950:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 07:59:17.876426 9189 raft_consensus.cc:1261] T 29322efc657745e8832e1d8ba1a88071 P 83f20e84af4c40d09a4804b6b26e697f [term 1 FOLLOWER]: Refusing update from remote peer 13d05ebdca8c4f489c597ce550dd773d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:17.876621 8794 raft_consensus.cc:1261] T 29322efc657745e8832e1d8ba1a88071 P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Refusing update from remote peer 13d05ebdca8c4f489c597ce550dd773d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:17.877024 13351 consensus_queue.cc:1035] T 29322efc657745e8832e1d8ba1a88071 P 13d05ebdca8c4f489c597ce550dd773d [LEADER]: Connected to new peer: Peer: permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:17.877178 13040 consensus_queue.cc:1035] T 29322efc657745e8832e1d8ba1a88071 P 13d05ebdca8c4f489c597ce550dd773d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:17.881189 6170 catalog_manager.cc:5576] TS 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133:39859) does not have the latest schema for tablet 29322efc657745e8832e1d8ba1a88071 (table rebalancer_dynamic_table [id=a4c8d5c1ec2c458da759b32ac1f778de]). Expected version 1 got 0
I20240501 07:59:17.881319 13394 tablet.cc:1722] T 29322efc657745e8832e1d8ba1a88071 P 13d05ebdca8c4f489c597ce550dd773d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 07:59:17.881712 6164 catalog_manager.cc:5576] TS 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134:40141) does not have the latest schema for tablet 29322efc657745e8832e1d8ba1a88071 (table rebalancer_dynamic_table [id=a4c8d5c1ec2c458da759b32ac1f778de]). Expected version 1 got 0
I20240501 07:59:17.881645 13394 tablet.cc:1620] T 29322efc657745e8832e1d8ba1a88071 P 13d05ebdca8c4f489c597ce550dd773d: MemRowSet was empty: no flush needed.
I20240501 07:59:17.884173 6170 catalog_manager.cc:5576] TS 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133:39859) does not have the latest schema for tablet 29322efc657745e8832e1d8ba1a88071 (table rebalancer_dynamic_table [id=a4c8d5c1ec2c458da759b32ac1f778de]). Expected version 1 got 0
I20240501 07:59:17.885408 6164 catalog_manager.cc:5576] TS 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149) does not have the latest schema for tablet 29322efc657745e8832e1d8ba1a88071 (table rebalancer_dynamic_table [id=a4c8d5c1ec2c458da759b32ac1f778de]). Expected version 1 got 0
I20240501 07:59:17.885490 13092 tablet.cc:1722] T 29322efc657745e8832e1d8ba1a88071 P 83f20e84af4c40d09a4804b6b26e697f: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 07:59:17.885833 13092 tablet.cc:1620] T 29322efc657745e8832e1d8ba1a88071 P 83f20e84af4c40d09a4804b6b26e697f: MemRowSet was empty: no flush needed.
I20240501 07:59:17.889459 13395 tablet.cc:1722] T 29322efc657745e8832e1d8ba1a88071 P 74077b61d6f54342bf9736b4a9f2eade: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 07:59:17.889912 13395 tablet.cc:1620] T 29322efc657745e8832e1d8ba1a88071 P 74077b61d6f54342bf9736b4a9f2eade: MemRowSet was empty: no flush needed.
I20240501 07:59:17.892215 6165 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 0bc706e85ce74cef97a4827ea4d4aa65: rebalancer_dynamic_table [id=a4c8d5c1ec2c458da759b32ac1f778de] alter complete (version 1)
I20240501 07:59:17.895280 13218 raft_consensus.cc:482] T fcb3198c1fa5449781dc84756fe02bff P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:17.895392 13218 raft_consensus.cc:504] T fcb3198c1fa5449781dc84756fe02bff P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } }
I20240501 07:59:17.895789 13218 leader_election.cc:288] T fcb3198c1fa5449781dc84756fe02bff P 74077b61d6f54342bf9736b4a9f2eade [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607), 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132:36773)
I20240501 07:59:17.896303 6707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcb3198c1fa5449781dc84756fe02bff" candidate_uuid: "74077b61d6f54342bf9736b4a9f2eade" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f9cb72be98941b5a8e504cbe34132b9" is_pre_election: true
I20240501 07:59:17.896317 9056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcb3198c1fa5449781dc84756fe02bff" candidate_uuid: "74077b61d6f54342bf9736b4a9f2eade" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a505e7cb133c4392bfda8a4a20874ae9" is_pre_election: true
I20240501 07:59:17.896456 9056 raft_consensus.cc:2454] T fcb3198c1fa5449781dc84756fe02bff P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74077b61d6f54342bf9736b4a9f2eade in term 0.
I20240501 07:59:17.896464 6707 raft_consensus.cc:2454] T fcb3198c1fa5449781dc84756fe02bff P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74077b61d6f54342bf9736b4a9f2eade in term 0.
I20240501 07:59:17.896730 8729 leader_election.cc:302] T fcb3198c1fa5449781dc84756fe02bff P 74077b61d6f54342bf9736b4a9f2eade [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f9cb72be98941b5a8e504cbe34132b9, 74077b61d6f54342bf9736b4a9f2eade; no voters:
I20240501 07:59:17.897091 13218 raft_consensus.cc:2781] T fcb3198c1fa5449781dc84756fe02bff P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:17.897186 13218 raft_consensus.cc:482] T fcb3198c1fa5449781dc84756fe02bff P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:17.897230 13218 raft_consensus.cc:3037] T fcb3198c1fa5449781dc84756fe02bff P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:17.898146 13218 raft_consensus.cc:504] T fcb3198c1fa5449781dc84756fe02bff P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } }
I20240501 07:59:17.898469 13218 leader_election.cc:288] T fcb3198c1fa5449781dc84756fe02bff P 74077b61d6f54342bf9736b4a9f2eade [CANDIDATE]: Term 1 election: Requested vote from peers a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607), 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132:36773)
I20240501 07:59:17.898798 9056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcb3198c1fa5449781dc84756fe02bff" candidate_uuid: "74077b61d6f54342bf9736b4a9f2eade" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a505e7cb133c4392bfda8a4a20874ae9"
I20240501 07:59:17.898823 6707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcb3198c1fa5449781dc84756fe02bff" candidate_uuid: "74077b61d6f54342bf9736b4a9f2eade" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f9cb72be98941b5a8e504cbe34132b9"
I20240501 07:59:17.898909 9056 raft_consensus.cc:3037] T fcb3198c1fa5449781dc84756fe02bff P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:17.898948 6707 raft_consensus.cc:3037] T fcb3198c1fa5449781dc84756fe02bff P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:17.899654 9056 raft_consensus.cc:2454] T fcb3198c1fa5449781dc84756fe02bff P a505e7cb133c4392bfda8a4a20874ae9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74077b61d6f54342bf9736b4a9f2eade in term 1.
I20240501 07:59:17.899916 8728 leader_election.cc:302] T fcb3198c1fa5449781dc84756fe02bff P 74077b61d6f54342bf9736b4a9f2eade [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74077b61d6f54342bf9736b4a9f2eade, a505e7cb133c4392bfda8a4a20874ae9; no voters:
I20240501 07:59:17.900077 13218 raft_consensus.cc:2781] T fcb3198c1fa5449781dc84756fe02bff P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:17.900065 6707 raft_consensus.cc:2454] T fcb3198c1fa5449781dc84756fe02bff P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74077b61d6f54342bf9736b4a9f2eade in term 1.
I20240501 07:59:17.900264 13218 raft_consensus.cc:686] T fcb3198c1fa5449781dc84756fe02bff P 74077b61d6f54342bf9736b4a9f2eade [term 1 LEADER]: Becoming Leader. State: Replica: 74077b61d6f54342bf9736b4a9f2eade, State: Running, Role: LEADER
I20240501 07:59:17.900422 13218 consensus_queue.cc:237] T fcb3198c1fa5449781dc84756fe02bff P 74077b61d6f54342bf9736b4a9f2eade [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } }
I20240501 07:59:17.902841 6165 catalog_manager.cc:5484] T fcb3198c1fa5449781dc84756fe02bff P 74077b61d6f54342bf9736b4a9f2eade reported cstate change: term changed from 0 to 1, leader changed from <none> to 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133). New cstate: current_term: 1 leader_uuid: "74077b61d6f54342bf9736b4a9f2eade" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } health_report { overall_health: HEALTHY } } }
I20240501 07:59:17.997102 6165 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55950:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 07:59:18.005162 13394 tablet.cc:1722] T 29322efc657745e8832e1d8ba1a88071 P 13d05ebdca8c4f489c597ce550dd773d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 07:59:18.005519 13394 tablet.cc:1620] T 29322efc657745e8832e1d8ba1a88071 P 13d05ebdca8c4f489c597ce550dd773d: MemRowSet was empty: no flush needed.
I20240501 07:59:18.005504 13092 tablet.cc:1722] T 29322efc657745e8832e1d8ba1a88071 P 83f20e84af4c40d09a4804b6b26e697f: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 07:59:18.005887 13092 tablet.cc:1620] T 29322efc657745e8832e1d8ba1a88071 P 83f20e84af4c40d09a4804b6b26e697f: MemRowSet was empty: no flush needed.
I20240501 07:59:18.011292 13395 tablet.cc:1722] T 29322efc657745e8832e1d8ba1a88071 P 74077b61d6f54342bf9736b4a9f2eade: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 07:59:18.011615 13395 tablet.cc:1620] T 29322efc657745e8832e1d8ba1a88071 P 74077b61d6f54342bf9736b4a9f2eade: MemRowSet was empty: no flush needed.
I20240501 07:59:18.015756 6165 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 0bc706e85ce74cef97a4827ea4d4aa65: rebalancer_dynamic_table [id=a4c8d5c1ec2c458da759b32ac1f778de] alter complete (version 2)
I20240501 07:59:18.019398 11901 raft_consensus.cc:1050] T 5ee738571af740c1b494d4f17413d1d8 P 8b8b3ad3aa8f4053b2167dd550f2086f: attempting to promote NON_VOTER 83f20e84af4c40d09a4804b6b26e697f to VOTER
I20240501 07:59:18.019979 11901 consensus_queue.cc:237] T 5ee738571af740c1b494d4f17413d1d8 P 8b8b3ad3aa8f4053b2167dd550f2086f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false } }
I20240501 07:59:18.022682 6577 raft_consensus.cc:1261] T 5ee738571af740c1b494d4f17413d1d8 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Refusing update from remote peer 8b8b3ad3aa8f4053b2167dd550f2086f: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 07:59:18.023226 12930 consensus_queue.cc:1035] T 5ee738571af740c1b494d4f17413d1d8 P 8b8b3ad3aa8f4053b2167dd550f2086f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 07:59:18.024415 8794 raft_consensus.cc:1261] T 5ee738571af740c1b494d4f17413d1d8 P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Refusing update from remote peer 8b8b3ad3aa8f4053b2167dd550f2086f: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 07:59:18.025465 9188 raft_consensus.cc:1261] T 5ee738571af740c1b494d4f17413d1d8 P 83f20e84af4c40d09a4804b6b26e697f [term 1 LEARNER]: Refusing update from remote peer 8b8b3ad3aa8f4053b2167dd550f2086f: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 07:59:18.025893 12930 consensus_queue.cc:1035] T 5ee738571af740c1b494d4f17413d1d8 P 8b8b3ad3aa8f4053b2167dd550f2086f [LEADER]: Connected to new peer: Peer: permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 07:59:18.027149 13304 consensus_queue.cc:1035] T 5ee738571af740c1b494d4f17413d1d8 P 8b8b3ad3aa8f4053b2167dd550f2086f [LEADER]: Connected to new peer: Peer: permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 07:59:18.028358 12930 raft_consensus.cc:2932] T 5ee738571af740c1b494d4f17413d1d8 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false } } }
I20240501 07:59:18.031160 6577 raft_consensus.cc:2932] T 5ee738571af740c1b494d4f17413d1d8 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false } } }
I20240501 07:59:18.032125 6165 catalog_manager.cc:5484] T 5ee738571af740c1b494d4f17413d1d8 P 8b8b3ad3aa8f4053b2167dd550f2086f reported cstate change: config changed from index 7 to 8, 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 07:59:18.034546 8794 raft_consensus.cc:2932] T 5ee738571af740c1b494d4f17413d1d8 P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false } } }
I20240501 07:59:18.037242 9188 raft_consensus.cc:2932] T 5ee738571af740c1b494d4f17413d1d8 P 83f20e84af4c40d09a4804b6b26e697f [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false } } }
I20240501 07:59:18.041615 6311 consensus_queue.cc:237] T 5ee738571af740c1b494d4f17413d1d8 P 8b8b3ad3aa8f4053b2167dd550f2086f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false } }
I20240501 07:59:18.042843 9188 raft_consensus.cc:1261] T 5ee738571af740c1b494d4f17413d1d8 P 83f20e84af4c40d09a4804b6b26e697f [term 1 FOLLOWER]: Refusing update from remote peer 8b8b3ad3aa8f4053b2167dd550f2086f: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 07:59:18.042842 8794 raft_consensus.cc:1261] T 5ee738571af740c1b494d4f17413d1d8 P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Refusing update from remote peer 8b8b3ad3aa8f4053b2167dd550f2086f: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 07:59:18.043258 13304 consensus_queue.cc:1035] T 5ee738571af740c1b494d4f17413d1d8 P 8b8b3ad3aa8f4053b2167dd550f2086f [LEADER]: Connected to new peer: Peer: permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 07:59:18.043586 13304 consensus_queue.cc:1035] T 5ee738571af740c1b494d4f17413d1d8 P 8b8b3ad3aa8f4053b2167dd550f2086f [LEADER]: Connected to new peer: Peer: permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 07:59:18.046701 13304 raft_consensus.cc:2932] T 5ee738571af740c1b494d4f17413d1d8 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false } } }
I20240501 07:59:18.049031 6157 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5ee738571af740c1b494d4f17413d1d8 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 07:59:18.050019 6165 catalog_manager.cc:5484] T 5ee738571af740c1b494d4f17413d1d8 P 8b8b3ad3aa8f4053b2167dd550f2086f reported cstate change: config changed from index 8 to 9, VOTER 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 07:59:18.051000 9188 raft_consensus.cc:2932] T 5ee738571af740c1b494d4f17413d1d8 P 83f20e84af4c40d09a4804b6b26e697f [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false } } }
I20240501 07:59:18.051136 8794 raft_consensus.cc:2932] T 5ee738571af740c1b494d4f17413d1d8 P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false } } }
I20240501 07:59:18.054347 6556 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ee738571af740c1b494d4f17413d1d8 with delete_type TABLET_DATA_TOMBSTONED (TS 3b164cbbc00f4e8eb3254ddd349a3ff9 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:53874
I20240501 07:59:18.054625 13303 tablet_replica.cc:330] T 5ee738571af740c1b494d4f17413d1d8 P 3b164cbbc00f4e8eb3254ddd349a3ff9: stopping tablet replica
I20240501 07:59:18.054795 13303 raft_consensus.cc:2229] T 5ee738571af740c1b494d4f17413d1d8 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:18.055052 13303 raft_consensus.cc:2258] T 5ee738571af740c1b494d4f17413d1d8 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:18.058306 13303 ts_tablet_manager.cc:1875] T 5ee738571af740c1b494d4f17413d1d8 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 07:59:18.061914 13303 ts_tablet_manager.cc:1888] T 5ee738571af740c1b494d4f17413d1d8 P 3b164cbbc00f4e8eb3254ddd349a3ff9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240501 07:59:18.062090 13303 log.cc:1196] T 5ee738571af740c1b494d4f17413d1d8 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/5ee738571af740c1b494d4f17413d1d8
I20240501 07:59:18.062886 6160 catalog_manager.cc:4830] TS 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131:34471): tablet 5ee738571af740c1b494d4f17413d1d8 (table rebalance_test_table_3 [id=eb6765f7643442ff82fde7f790b2aaed]) successfully deleted
I20240501 07:59:18.122840 13297 consensus_queue.cc:1035] T 9461ad0f65e1489ca102f2558194ce8f P 8315d22c56f7439a9e1fc82f153f018c [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:18.125051 6170 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55950:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 07:59:18.131601 13172 consensus_queue.cc:1035] T 9461ad0f65e1489ca102f2558194ce8f P 8315d22c56f7439a9e1fc82f153f018c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:18.132303 13394 tablet.cc:1722] T 29322efc657745e8832e1d8ba1a88071 P 13d05ebdca8c4f489c597ce550dd773d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 07:59:18.134316 13395 tablet.cc:1722] T 29322efc657745e8832e1d8ba1a88071 P 74077b61d6f54342bf9736b4a9f2eade: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 07:59:18.139957 13092 tablet.cc:1722] T 29322efc657745e8832e1d8ba1a88071 P 83f20e84af4c40d09a4804b6b26e697f: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 07:59:18.144515 6165 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 0bc706e85ce74cef97a4827ea4d4aa65: rebalancer_dynamic_table_new_name [id=a4c8d5c1ec2c458da759b32ac1f778de] alter complete (version 3)
I20240501 07:59:18.248296 6171 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55950:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 07:59:18.248435 6171 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55950:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 07:59:18.250700 6171 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 0bc706e85ce74cef97a4827ea4d4aa65: Sending DeleteTablet for 3 replicas of tablet 29322efc657745e8832e1d8ba1a88071
I20240501 07:59:18.251482 8774 tablet_service.cc:1514] Processing DeleteTablet for tablet 29322efc657745e8832e1d8ba1a88071 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:18 UTC) from {username='slave'} at 127.0.0.1:43136
I20240501 07:59:18.251664 8905 tablet_service.cc:1514] Processing DeleteTablet for tablet 29322efc657745e8832e1d8ba1a88071 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:18 UTC) from {username='slave'} at 127.0.0.1:47202
I20240501 07:59:18.251941 9167 tablet_service.cc:1514] Processing DeleteTablet for tablet 29322efc657745e8832e1d8ba1a88071 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:18 UTC) from {username='slave'} at 127.0.0.1:33594
I20240501 07:59:18.252244 13472 tablet_replica.cc:330] T 29322efc657745e8832e1d8ba1a88071 P 83f20e84af4c40d09a4804b6b26e697f: stopping tablet replica
I20240501 07:59:18.252440 13472 raft_consensus.cc:2229] T 29322efc657745e8832e1d8ba1a88071 P 83f20e84af4c40d09a4804b6b26e697f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:18.252606 13472 raft_consensus.cc:2258] T 29322efc657745e8832e1d8ba1a88071 P 83f20e84af4c40d09a4804b6b26e697f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:18.253438 13472 ts_tablet_manager.cc:1875] T 29322efc657745e8832e1d8ba1a88071 P 83f20e84af4c40d09a4804b6b26e697f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:59:18.256279 13472 ts_tablet_manager.cc:1888] T 29322efc657745e8832e1d8ba1a88071 P 83f20e84af4c40d09a4804b6b26e697f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:59:18.256405 13472 log.cc:1196] T 29322efc657745e8832e1d8ba1a88071 P 83f20e84af4c40d09a4804b6b26e697f: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/29322efc657745e8832e1d8ba1a88071
I20240501 07:59:18.256680 13472 ts_tablet_manager.cc:1909] T 29322efc657745e8832e1d8ba1a88071 P 83f20e84af4c40d09a4804b6b26e697f: Deleting consensus metadata
I20240501 07:59:18.257366 6158 catalog_manager.cc:4830] TS 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): tablet 29322efc657745e8832e1d8ba1a88071 (table rebalancer_dynamic_table_new_name [id=a4c8d5c1ec2c458da759b32ac1f778de]) successfully deleted
I20240501 07:59:18.258800 13470 tablet_replica.cc:330] T 29322efc657745e8832e1d8ba1a88071 P 74077b61d6f54342bf9736b4a9f2eade: stopping tablet replica
I20240501 07:59:18.259091 13471 tablet_replica.cc:330] T 29322efc657745e8832e1d8ba1a88071 P 13d05ebdca8c4f489c597ce550dd773d: stopping tablet replica
I20240501 07:59:18.259299 13471 raft_consensus.cc:2229] T 29322efc657745e8832e1d8ba1a88071 P 13d05ebdca8c4f489c597ce550dd773d [term 1 LEADER]: Raft consensus shutting down.
I20240501 07:59:18.259541 13471 raft_consensus.cc:2258] T 29322efc657745e8832e1d8ba1a88071 P 13d05ebdca8c4f489c597ce550dd773d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:18.259799 13470 raft_consensus.cc:2229] T 29322efc657745e8832e1d8ba1a88071 P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:18.259989 13470 raft_consensus.cc:2258] T 29322efc657745e8832e1d8ba1a88071 P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:18.261430 13470 ts_tablet_manager.cc:1875] T 29322efc657745e8832e1d8ba1a88071 P 74077b61d6f54342bf9736b4a9f2eade: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:59:18.262264 13471 ts_tablet_manager.cc:1875] T 29322efc657745e8832e1d8ba1a88071 P 13d05ebdca8c4f489c597ce550dd773d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:59:18.264562 13471 ts_tablet_manager.cc:1888] T 29322efc657745e8832e1d8ba1a88071 P 13d05ebdca8c4f489c597ce550dd773d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:59:18.264636 13471 log.cc:1196] T 29322efc657745e8832e1d8ba1a88071 P 13d05ebdca8c4f489c597ce550dd773d: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/29322efc657745e8832e1d8ba1a88071
I20240501 07:59:18.264901 13471 ts_tablet_manager.cc:1909] T 29322efc657745e8832e1d8ba1a88071 P 13d05ebdca8c4f489c597ce550dd773d: Deleting consensus metadata
I20240501 07:59:18.267405 13470 ts_tablet_manager.cc:1888] T 29322efc657745e8832e1d8ba1a88071 P 74077b61d6f54342bf9736b4a9f2eade: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:59:18.267482 13470 log.cc:1196] T 29322efc657745e8832e1d8ba1a88071 P 74077b61d6f54342bf9736b4a9f2eade: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/29322efc657745e8832e1d8ba1a88071
I20240501 07:59:18.267753 13470 ts_tablet_manager.cc:1909] T 29322efc657745e8832e1d8ba1a88071 P 74077b61d6f54342bf9736b4a9f2eade: Deleting consensus metadata
I20240501 07:59:18.268414 13266 consensus_queue.cc:1035] T 4d4f47308d174a92bc73f23f2242be9a P 83f20e84af4c40d09a4804b6b26e697f [LEADER]: Connected to new peer: Peer: permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:18.268653 6157 catalog_manager.cc:4830] TS 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133:39859): tablet 29322efc657745e8832e1d8ba1a88071 (table rebalancer_dynamic_table_new_name [id=a4c8d5c1ec2c458da759b32ac1f778de]) successfully deleted
I20240501 07:59:18.269085 6159 catalog_manager.cc:4830] TS 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134:40141): tablet 29322efc657745e8832e1d8ba1a88071 (table rebalancer_dynamic_table_new_name [id=a4c8d5c1ec2c458da759b32ac1f778de]) successfully deleted
I20240501 07:59:18.281388 13383 consensus_queue.cc:1035] T 4d4f47308d174a92bc73f23f2242be9a P 83f20e84af4c40d09a4804b6b26e697f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:18.284261 13218 consensus_queue.cc:1035] T fcb3198c1fa5449781dc84756fe02bff P 74077b61d6f54342bf9736b4a9f2eade [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:18.292974 13396 consensus_queue.cc:1035] T fcb3198c1fa5449781dc84756fe02bff P 74077b61d6f54342bf9736b4a9f2eade [LEADER]: Connected to new peer: Peer: permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:18.337747 8774 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 07:59:18.340258 8905 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 07:59:18.340551 6687 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 07:59:18.345074 6556 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 07:59:18.352454 6424 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 07:59:18.357182 6164 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55950:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 07:59:18.359995 9167 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 07:59:18.369669 8774 tablet_service.cc:1467] Processing CreateTablet for tablet 74d6caf4bbf345a0b7443780afd7fc01 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=d23a3938211c43b4b7b7a3b54b1b6beb]), partition=
I20240501 07:59:18.369964 8774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74d6caf4bbf345a0b7443780afd7fc01. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:18.374218 9167 tablet_service.cc:1467] Processing CreateTablet for tablet 74d6caf4bbf345a0b7443780afd7fc01 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=d23a3938211c43b4b7b7a3b54b1b6beb]), partition=
I20240501 07:59:18.374495 9167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74d6caf4bbf345a0b7443780afd7fc01. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:18.375556 6556 tablet_service.cc:1467] Processing CreateTablet for tablet 74d6caf4bbf345a0b7443780afd7fc01 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=d23a3938211c43b4b7b7a3b54b1b6beb]), partition=
I20240501 07:59:18.375782 6556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74d6caf4bbf345a0b7443780afd7fc01. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:18.376775 6295 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 07:59:18.387754 13497 tablet_bootstrap.cc:492] T 74d6caf4bbf345a0b7443780afd7fc01 P 83f20e84af4c40d09a4804b6b26e697f: Bootstrap starting.
I20240501 07:59:18.388510 9036 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 07:59:18.397830 13497 tablet_bootstrap.cc:654] T 74d6caf4bbf345a0b7443780afd7fc01 P 83f20e84af4c40d09a4804b6b26e697f: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:18.400857 13497 tablet_bootstrap.cc:492] T 74d6caf4bbf345a0b7443780afd7fc01 P 83f20e84af4c40d09a4804b6b26e697f: No bootstrap required, opened a new log
I20240501 07:59:18.401082 13497 ts_tablet_manager.cc:1367] T 74d6caf4bbf345a0b7443780afd7fc01 P 83f20e84af4c40d09a4804b6b26e697f: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.000s
I20240501 07:59:18.401656 13497 raft_consensus.cc:348] T 74d6caf4bbf345a0b7443780afd7fc01 P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } }
I20240501 07:59:18.401870 13498 tablet_bootstrap.cc:492] T 74d6caf4bbf345a0b7443780afd7fc01 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Bootstrap starting.
I20240501 07:59:18.401937 13497 raft_consensus.cc:374] T 74d6caf4bbf345a0b7443780afd7fc01 P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:18.402024 13497 raft_consensus.cc:729] T 74d6caf4bbf345a0b7443780afd7fc01 P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83f20e84af4c40d09a4804b6b26e697f, State: Initialized, Role: FOLLOWER
I20240501 07:59:18.402108 13496 tablet_bootstrap.cc:492] T 74d6caf4bbf345a0b7443780afd7fc01 P 74077b61d6f54342bf9736b4a9f2eade: Bootstrap starting.
I20240501 07:59:18.402274 13497 consensus_queue.cc:260] T 74d6caf4bbf345a0b7443780afd7fc01 P 83f20e84af4c40d09a4804b6b26e697f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } }
I20240501 07:59:18.403210 13496 tablet_bootstrap.cc:654] T 74d6caf4bbf345a0b7443780afd7fc01 P 74077b61d6f54342bf9736b4a9f2eade: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:18.403661 13497 ts_tablet_manager.cc:1398] T 74d6caf4bbf345a0b7443780afd7fc01 P 83f20e84af4c40d09a4804b6b26e697f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:59:18.405426 13498 tablet_bootstrap.cc:654] T 74d6caf4bbf345a0b7443780afd7fc01 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:18.406445 13496 tablet_bootstrap.cc:492] T 74d6caf4bbf345a0b7443780afd7fc01 P 74077b61d6f54342bf9736b4a9f2eade: No bootstrap required, opened a new log
I20240501 07:59:18.406733 13496 ts_tablet_manager.cc:1367] T 74d6caf4bbf345a0b7443780afd7fc01 P 74077b61d6f54342bf9736b4a9f2eade: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 07:59:18.407356 13496 raft_consensus.cc:348] T 74d6caf4bbf345a0b7443780afd7fc01 P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } }
I20240501 07:59:18.407544 13496 raft_consensus.cc:374] T 74d6caf4bbf345a0b7443780afd7fc01 P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:18.407629 13496 raft_consensus.cc:729] T 74d6caf4bbf345a0b7443780afd7fc01 P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74077b61d6f54342bf9736b4a9f2eade, State: Initialized, Role: FOLLOWER
I20240501 07:59:18.407860 13496 consensus_queue.cc:260] T 74d6caf4bbf345a0b7443780afd7fc01 P 74077b61d6f54342bf9736b4a9f2eade [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } }
I20240501 07:59:18.409519 13496 ts_tablet_manager.cc:1398] T 74d6caf4bbf345a0b7443780afd7fc01 P 74077b61d6f54342bf9736b4a9f2eade: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 07:59:18.410282 13498 tablet_bootstrap.cc:492] T 74d6caf4bbf345a0b7443780afd7fc01 P 3b164cbbc00f4e8eb3254ddd349a3ff9: No bootstrap required, opened a new log
I20240501 07:59:18.411686 6171 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55950:
name: "rebalancer_extra_table_28"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 07:59:18.412364 13498 ts_tablet_manager.cc:1367] T 74d6caf4bbf345a0b7443780afd7fc01 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.000s
I20240501 07:59:18.412860 13498 raft_consensus.cc:348] T 74d6caf4bbf345a0b7443780afd7fc01 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } }
I20240501 07:59:18.412993 13498 raft_consensus.cc:374] T 74d6caf4bbf345a0b7443780afd7fc01 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:18.413023 13498 raft_consensus.cc:729] T 74d6caf4bbf345a0b7443780afd7fc01 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b164cbbc00f4e8eb3254ddd349a3ff9, State: Initialized, Role: FOLLOWER
I20240501 07:59:18.413130 13498 consensus_queue.cc:260] T 74d6caf4bbf345a0b7443780afd7fc01 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } }
I20240501 07:59:18.414330 13498 ts_tablet_manager.cc:1398] T 74d6caf4bbf345a0b7443780afd7fc01 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:59:18.421151 8774 tablet_service.cc:1467] Processing CreateTablet for tablet f0c1c05110894e39b41e6b5c5894b8d8 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=355ca58af7de4b229e5afd82e037bec6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:18.421512 8774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0c1c05110894e39b41e6b5c5894b8d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:18.421953 9167 tablet_service.cc:1467] Processing CreateTablet for tablet f0c1c05110894e39b41e6b5c5894b8d8 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=355ca58af7de4b229e5afd82e037bec6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:18.422235 9167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0c1c05110894e39b41e6b5c5894b8d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:18.422406 6424 tablet_service.cc:1467] Processing CreateTablet for tablet bf806039d8c742d8a80ad994829bdf18 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=355ca58af7de4b229e5afd82e037bec6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:18.422699 6424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf806039d8c742d8a80ad994829bdf18. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:18.423869 13496 tablet_bootstrap.cc:492] T f0c1c05110894e39b41e6b5c5894b8d8 P 74077b61d6f54342bf9736b4a9f2eade: Bootstrap starting.
I20240501 07:59:18.424131 6556 tablet_service.cc:1467] Processing CreateTablet for tablet af718dc43add4edfac703013655699f4 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=355ca58af7de4b229e5afd82e037bec6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:18.424397 6556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af718dc43add4edfac703013655699f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:18.424546 9036 tablet_service.cc:1467] Processing CreateTablet for tablet af718dc43add4edfac703013655699f4 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=355ca58af7de4b229e5afd82e037bec6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:18.424823 9036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af718dc43add4edfac703013655699f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:18.425101 13496 tablet_bootstrap.cc:654] T f0c1c05110894e39b41e6b5c5894b8d8 P 74077b61d6f54342bf9736b4a9f2eade: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:18.425137 13497 tablet_bootstrap.cc:492] T f0c1c05110894e39b41e6b5c5894b8d8 P 83f20e84af4c40d09a4804b6b26e697f: Bootstrap starting.
I20240501 07:59:18.425807 13502 tablet_bootstrap.cc:492] T bf806039d8c742d8a80ad994829bdf18 P 8315d22c56f7439a9e1fc82f153f018c: Bootstrap starting.
I20240501 07:59:18.425999 8905 tablet_service.cc:1467] Processing CreateTablet for tablet f0c1c05110894e39b41e6b5c5894b8d8 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=355ca58af7de4b229e5afd82e037bec6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:18.426137 13497 tablet_bootstrap.cc:654] T f0c1c05110894e39b41e6b5c5894b8d8 P 83f20e84af4c40d09a4804b6b26e697f: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:18.426281 8905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0c1c05110894e39b41e6b5c5894b8d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:18.428342 8774 tablet_service.cc:1467] Processing CreateTablet for tablet af718dc43add4edfac703013655699f4 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=355ca58af7de4b229e5afd82e037bec6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:18.428635 8774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af718dc43add4edfac703013655699f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:18.429337 13506 tablet_bootstrap.cc:492] T af718dc43add4edfac703013655699f4 P a505e7cb133c4392bfda8a4a20874ae9: Bootstrap starting.
I20240501 07:59:18.430959 9036 tablet_service.cc:1467] Processing CreateTablet for tablet bf806039d8c742d8a80ad994829bdf18 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=355ca58af7de4b229e5afd82e037bec6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:18.431210 9036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf806039d8c742d8a80ad994829bdf18. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:18.432040 13498 tablet_bootstrap.cc:492] T af718dc43add4edfac703013655699f4 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Bootstrap starting.
I20240501 07:59:18.432693 13502 tablet_bootstrap.cc:654] T bf806039d8c742d8a80ad994829bdf18 P 8315d22c56f7439a9e1fc82f153f018c: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:18.433406 9167 tablet_service.cc:1467] Processing CreateTablet for tablet bf806039d8c742d8a80ad994829bdf18 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=355ca58af7de4b229e5afd82e037bec6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:18.433601 13498 tablet_bootstrap.cc:654] T af718dc43add4edfac703013655699f4 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:18.433693 9167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf806039d8c742d8a80ad994829bdf18. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:18.434150 13496 tablet_bootstrap.cc:492] T f0c1c05110894e39b41e6b5c5894b8d8 P 74077b61d6f54342bf9736b4a9f2eade: No bootstrap required, opened a new log
I20240501 07:59:18.434244 13496 ts_tablet_manager.cc:1367] T f0c1c05110894e39b41e6b5c5894b8d8 P 74077b61d6f54342bf9736b4a9f2eade: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.001s
I20240501 07:59:18.434295 13497 tablet_bootstrap.cc:492] T f0c1c05110894e39b41e6b5c5894b8d8 P 83f20e84af4c40d09a4804b6b26e697f: No bootstrap required, opened a new log
I20240501 07:59:18.434373 13497 ts_tablet_manager.cc:1367] T f0c1c05110894e39b41e6b5c5894b8d8 P 83f20e84af4c40d09a4804b6b26e697f: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240501 07:59:18.434688 13496 raft_consensus.cc:348] T f0c1c05110894e39b41e6b5c5894b8d8 P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } }
I20240501 07:59:18.434798 13496 raft_consensus.cc:374] T f0c1c05110894e39b41e6b5c5894b8d8 P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:18.434826 13496 raft_consensus.cc:729] T f0c1c05110894e39b41e6b5c5894b8d8 P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74077b61d6f54342bf9736b4a9f2eade, State: Initialized, Role: FOLLOWER
I20240501 07:59:18.434854 13497 raft_consensus.cc:348] T f0c1c05110894e39b41e6b5c5894b8d8 P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } }
I20240501 07:59:18.434952 13497 raft_consensus.cc:374] T f0c1c05110894e39b41e6b5c5894b8d8 P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:18.434942 13496 consensus_queue.cc:260] T f0c1c05110894e39b41e6b5c5894b8d8 P 74077b61d6f54342bf9736b4a9f2eade [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } }
I20240501 07:59:18.434989 13497 raft_consensus.cc:729] T f0c1c05110894e39b41e6b5c5894b8d8 P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83f20e84af4c40d09a4804b6b26e697f, State: Initialized, Role: FOLLOWER
I20240501 07:59:18.435110 13497 consensus_queue.cc:260] T f0c1c05110894e39b41e6b5c5894b8d8 P 83f20e84af4c40d09a4804b6b26e697f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } }
I20240501 07:59:18.435214 13496 ts_tablet_manager.cc:1398] T f0c1c05110894e39b41e6b5c5894b8d8 P 74077b61d6f54342bf9736b4a9f2eade: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 07:59:18.435396 13496 tablet_bootstrap.cc:492] T af718dc43add4edfac703013655699f4 P 74077b61d6f54342bf9736b4a9f2eade: Bootstrap starting.
I20240501 07:59:18.435388 13497 ts_tablet_manager.cc:1398] T f0c1c05110894e39b41e6b5c5894b8d8 P 83f20e84af4c40d09a4804b6b26e697f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:18.436314 13496 tablet_bootstrap.cc:654] T af718dc43add4edfac703013655699f4 P 74077b61d6f54342bf9736b4a9f2eade: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:18.436458 13497 tablet_bootstrap.cc:492] T bf806039d8c742d8a80ad994829bdf18 P 83f20e84af4c40d09a4804b6b26e697f: Bootstrap starting.
I20240501 07:59:18.436945 13502 tablet_bootstrap.cc:492] T bf806039d8c742d8a80ad994829bdf18 P 8315d22c56f7439a9e1fc82f153f018c: No bootstrap required, opened a new log
I20240501 07:59:18.437089 13502 ts_tablet_manager.cc:1367] T bf806039d8c742d8a80ad994829bdf18 P 8315d22c56f7439a9e1fc82f153f018c: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.001s
I20240501 07:59:18.437402 13497 tablet_bootstrap.cc:654] T bf806039d8c742d8a80ad994829bdf18 P 83f20e84af4c40d09a4804b6b26e697f: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:18.437691 13505 tablet_bootstrap.cc:492] T f0c1c05110894e39b41e6b5c5894b8d8 P 13d05ebdca8c4f489c597ce550dd773d: Bootstrap starting.
I20240501 07:59:18.437641 13502 raft_consensus.cc:348] T bf806039d8c742d8a80ad994829bdf18 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } }
I20240501 07:59:18.437763 13502 raft_consensus.cc:374] T bf806039d8c742d8a80ad994829bdf18 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:18.437799 13502 raft_consensus.cc:729] T bf806039d8c742d8a80ad994829bdf18 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8315d22c56f7439a9e1fc82f153f018c, State: Initialized, Role: FOLLOWER
I20240501 07:59:18.437932 13502 consensus_queue.cc:260] T bf806039d8c742d8a80ad994829bdf18 P 8315d22c56f7439a9e1fc82f153f018c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } }
I20240501 07:59:18.438755 13505 tablet_bootstrap.cc:654] T f0c1c05110894e39b41e6b5c5894b8d8 P 13d05ebdca8c4f489c597ce550dd773d: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:18.430661 13506 tablet_bootstrap.cc:654] T af718dc43add4edfac703013655699f4 P a505e7cb133c4392bfda8a4a20874ae9: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:18.439841 13496 tablet_bootstrap.cc:492] T af718dc43add4edfac703013655699f4 P 74077b61d6f54342bf9736b4a9f2eade: No bootstrap required, opened a new log
I20240501 07:59:18.439922 13496 ts_tablet_manager.cc:1367] T af718dc43add4edfac703013655699f4 P 74077b61d6f54342bf9736b4a9f2eade: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20240501 07:59:18.440341 13496 raft_consensus.cc:348] T af718dc43add4edfac703013655699f4 P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } }
I20240501 07:59:18.440443 13496 raft_consensus.cc:374] T af718dc43add4edfac703013655699f4 P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:18.440474 13496 raft_consensus.cc:729] T af718dc43add4edfac703013655699f4 P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74077b61d6f54342bf9736b4a9f2eade, State: Initialized, Role: FOLLOWER
I20240501 07:59:18.440629 13497 tablet_bootstrap.cc:492] T bf806039d8c742d8a80ad994829bdf18 P 83f20e84af4c40d09a4804b6b26e697f: No bootstrap required, opened a new log
I20240501 07:59:18.440738 13497 ts_tablet_manager.cc:1367] T bf806039d8c742d8a80ad994829bdf18 P 83f20e84af4c40d09a4804b6b26e697f: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 07:59:18.441037 13496 consensus_queue.cc:260] T af718dc43add4edfac703013655699f4 P 74077b61d6f54342bf9736b4a9f2eade [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } }
I20240501 07:59:18.441527 13496 ts_tablet_manager.cc:1398] T af718dc43add4edfac703013655699f4 P 74077b61d6f54342bf9736b4a9f2eade: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 07:59:18.441900 13497 raft_consensus.cc:348] T bf806039d8c742d8a80ad994829bdf18 P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } }
I20240501 07:59:18.442010 13497 raft_consensus.cc:374] T bf806039d8c742d8a80ad994829bdf18 P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:18.442049 13497 raft_consensus.cc:729] T bf806039d8c742d8a80ad994829bdf18 P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83f20e84af4c40d09a4804b6b26e697f, State: Initialized, Role: FOLLOWER
I20240501 07:59:18.442139 13498 tablet_bootstrap.cc:492] T af718dc43add4edfac703013655699f4 P 3b164cbbc00f4e8eb3254ddd349a3ff9: No bootstrap required, opened a new log
I20240501 07:59:18.442163 13497 consensus_queue.cc:260] T bf806039d8c742d8a80ad994829bdf18 P 83f20e84af4c40d09a4804b6b26e697f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } }
I20240501 07:59:18.442234 13498 ts_tablet_manager.cc:1367] T af718dc43add4edfac703013655699f4 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.002s
I20240501 07:59:18.442345 13505 tablet_bootstrap.cc:492] T f0c1c05110894e39b41e6b5c5894b8d8 P 13d05ebdca8c4f489c597ce550dd773d: No bootstrap required, opened a new log
I20240501 07:59:18.442425 13505 ts_tablet_manager.cc:1367] T f0c1c05110894e39b41e6b5c5894b8d8 P 13d05ebdca8c4f489c597ce550dd773d: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 07:59:18.442463 13497 ts_tablet_manager.cc:1398] T bf806039d8c742d8a80ad994829bdf18 P 83f20e84af4c40d09a4804b6b26e697f: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 07:59:18.442708 13498 raft_consensus.cc:348] T af718dc43add4edfac703013655699f4 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } }
I20240501 07:59:18.442921 13498 raft_consensus.cc:374] T af718dc43add4edfac703013655699f4 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:18.442987 13498 raft_consensus.cc:729] T af718dc43add4edfac703013655699f4 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b164cbbc00f4e8eb3254ddd349a3ff9, State: Initialized, Role: FOLLOWER
I20240501 07:59:18.442871 13505 raft_consensus.cc:348] T f0c1c05110894e39b41e6b5c5894b8d8 P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } }
I20240501 07:59:18.443080 13505 raft_consensus.cc:374] T f0c1c05110894e39b41e6b5c5894b8d8 P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:18.443174 13505 raft_consensus.cc:729] T f0c1c05110894e39b41e6b5c5894b8d8 P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13d05ebdca8c4f489c597ce550dd773d, State: Initialized, Role: FOLLOWER
I20240501 07:59:18.443146 13498 consensus_queue.cc:260] T af718dc43add4edfac703013655699f4 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } }
I20240501 07:59:18.443333 13505 consensus_queue.cc:260] T f0c1c05110894e39b41e6b5c5894b8d8 P 13d05ebdca8c4f489c597ce550dd773d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } }
I20240501 07:59:18.444324 13502 ts_tablet_manager.cc:1398] T bf806039d8c742d8a80ad994829bdf18 P 8315d22c56f7439a9e1fc82f153f018c: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20240501 07:59:18.447907 13498 ts_tablet_manager.cc:1398] T af718dc43add4edfac703013655699f4 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Time spent starting tablet: real 0.006s user 0.000s sys 0.001s
I20240501 07:59:18.448179 13505 ts_tablet_manager.cc:1398] T f0c1c05110894e39b41e6b5c5894b8d8 P 13d05ebdca8c4f489c597ce550dd773d: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20240501 07:59:18.453682 13396 raft_consensus.cc:482] T 74d6caf4bbf345a0b7443780afd7fc01 P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:18.453812 13396 raft_consensus.cc:504] T 74d6caf4bbf345a0b7443780afd7fc01 P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } }
I20240501 07:59:18.454219 13396 leader_election.cc:288] T 74d6caf4bbf345a0b7443780afd7fc01 P 74077b61d6f54342bf9736b4a9f2eade [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131:34471), 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149)
I20240501 07:59:18.454609 6577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74d6caf4bbf345a0b7443780afd7fc01" candidate_uuid: "74077b61d6f54342bf9736b4a9f2eade" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" is_pre_election: true
I20240501 07:59:18.454725 6577 raft_consensus.cc:2454] T 74d6caf4bbf345a0b7443780afd7fc01 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74077b61d6f54342bf9736b4a9f2eade in term 0.
I20240501 07:59:18.455375 13506 tablet_bootstrap.cc:492] T af718dc43add4edfac703013655699f4 P a505e7cb133c4392bfda8a4a20874ae9: No bootstrap required, opened a new log
I20240501 07:59:18.455574 13506 ts_tablet_manager.cc:1367] T af718dc43add4edfac703013655699f4 P a505e7cb133c4392bfda8a4a20874ae9: Time spent bootstrapping tablet: real 0.026s user 0.002s sys 0.000s
I20240501 07:59:18.455693 9188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74d6caf4bbf345a0b7443780afd7fc01" candidate_uuid: "74077b61d6f54342bf9736b4a9f2eade" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83f20e84af4c40d09a4804b6b26e697f" is_pre_election: true
I20240501 07:59:18.455847 9188 raft_consensus.cc:2454] T 74d6caf4bbf345a0b7443780afd7fc01 P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74077b61d6f54342bf9736b4a9f2eade in term 0.
I20240501 07:59:18.456167 8730 leader_election.cc:302] T 74d6caf4bbf345a0b7443780afd7fc01 P 74077b61d6f54342bf9736b4a9f2eade [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b164cbbc00f4e8eb3254ddd349a3ff9, 74077b61d6f54342bf9736b4a9f2eade; no voters:
I20240501 07:59:18.456658 13396 raft_consensus.cc:2781] T 74d6caf4bbf345a0b7443780afd7fc01 P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:18.456743 13396 raft_consensus.cc:482] T 74d6caf4bbf345a0b7443780afd7fc01 P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:18.456809 13396 raft_consensus.cc:3037] T 74d6caf4bbf345a0b7443780afd7fc01 P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:18.458014 13396 raft_consensus.cc:504] T 74d6caf4bbf345a0b7443780afd7fc01 P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } }
I20240501 07:59:18.458405 13396 leader_election.cc:288] T 74d6caf4bbf345a0b7443780afd7fc01 P 74077b61d6f54342bf9736b4a9f2eade [CANDIDATE]: Term 1 election: Requested vote from peers 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131:34471), 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149)
I20240501 07:59:18.458361 13506 raft_consensus.cc:348] T af718dc43add4edfac703013655699f4 P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } }
I20240501 07:59:18.458480 13506 raft_consensus.cc:374] T af718dc43add4edfac703013655699f4 P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:18.458518 13506 raft_consensus.cc:729] T af718dc43add4edfac703013655699f4 P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a505e7cb133c4392bfda8a4a20874ae9, State: Initialized, Role: FOLLOWER
I20240501 07:59:18.458681 13506 consensus_queue.cc:260] T af718dc43add4edfac703013655699f4 P a505e7cb133c4392bfda8a4a20874ae9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } }
I20240501 07:59:18.458736 6577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74d6caf4bbf345a0b7443780afd7fc01" candidate_uuid: "74077b61d6f54342bf9736b4a9f2eade" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9"
I20240501 07:59:18.458873 6577 raft_consensus.cc:3037] T 74d6caf4bbf345a0b7443780afd7fc01 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:18.459137 13506 ts_tablet_manager.cc:1398] T af718dc43add4edfac703013655699f4 P a505e7cb133c4392bfda8a4a20874ae9: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 07:59:18.459662 9188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74d6caf4bbf345a0b7443780afd7fc01" candidate_uuid: "74077b61d6f54342bf9736b4a9f2eade" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83f20e84af4c40d09a4804b6b26e697f"
I20240501 07:59:18.459795 9188 raft_consensus.cc:3037] T 74d6caf4bbf345a0b7443780afd7fc01 P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:18.459933 6577 raft_consensus.cc:2454] T 74d6caf4bbf345a0b7443780afd7fc01 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74077b61d6f54342bf9736b4a9f2eade in term 1.
I20240501 07:59:18.460718 9188 raft_consensus.cc:2454] T 74d6caf4bbf345a0b7443780afd7fc01 P 83f20e84af4c40d09a4804b6b26e697f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74077b61d6f54342bf9736b4a9f2eade in term 1.
I20240501 07:59:18.461140 8728 leader_election.cc:302] T 74d6caf4bbf345a0b7443780afd7fc01 P 74077b61d6f54342bf9736b4a9f2eade [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74077b61d6f54342bf9736b4a9f2eade, 83f20e84af4c40d09a4804b6b26e697f; no voters:
I20240501 07:59:18.462013 13396 raft_consensus.cc:2781] T 74d6caf4bbf345a0b7443780afd7fc01 P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:18.462124 13396 raft_consensus.cc:686] T 74d6caf4bbf345a0b7443780afd7fc01 P 74077b61d6f54342bf9736b4a9f2eade [term 1 LEADER]: Becoming Leader. State: Replica: 74077b61d6f54342bf9736b4a9f2eade, State: Running, Role: LEADER
I20240501 07:59:18.462270 13396 consensus_queue.cc:237] T 74d6caf4bbf345a0b7443780afd7fc01 P 74077b61d6f54342bf9736b4a9f2eade [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } }
I20240501 07:59:18.464790 13506 tablet_bootstrap.cc:492] T bf806039d8c742d8a80ad994829bdf18 P a505e7cb133c4392bfda8a4a20874ae9: Bootstrap starting.
I20240501 07:59:18.464716 6168 catalog_manager.cc:5484] T 74d6caf4bbf345a0b7443780afd7fc01 P 74077b61d6f54342bf9736b4a9f2eade reported cstate change: term changed from 0 to 1, leader changed from <none> to 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133). New cstate: current_term: 1 leader_uuid: "74077b61d6f54342bf9736b4a9f2eade" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } health_report { overall_health: UNKNOWN } } }
I20240501 07:59:18.467922 13506 tablet_bootstrap.cc:654] T bf806039d8c742d8a80ad994829bdf18 P a505e7cb133c4392bfda8a4a20874ae9: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:18.469280 13506 tablet_bootstrap.cc:492] T bf806039d8c742d8a80ad994829bdf18 P a505e7cb133c4392bfda8a4a20874ae9: No bootstrap required, opened a new log
I20240501 07:59:18.469372 13506 ts_tablet_manager.cc:1367] T bf806039d8c742d8a80ad994829bdf18 P a505e7cb133c4392bfda8a4a20874ae9: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 07:59:18.469838 13506 raft_consensus.cc:348] T bf806039d8c742d8a80ad994829bdf18 P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } }
I20240501 07:59:18.469962 13506 raft_consensus.cc:374] T bf806039d8c742d8a80ad994829bdf18 P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:18.470024 13506 raft_consensus.cc:729] T bf806039d8c742d8a80ad994829bdf18 P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a505e7cb133c4392bfda8a4a20874ae9, State: Initialized, Role: FOLLOWER
I20240501 07:59:18.470151 13506 consensus_queue.cc:260] T bf806039d8c742d8a80ad994829bdf18 P a505e7cb133c4392bfda8a4a20874ae9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } }
I20240501 07:59:18.470434 13506 ts_tablet_manager.cc:1398] T bf806039d8c742d8a80ad994829bdf18 P a505e7cb133c4392bfda8a4a20874ae9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:18.485227 9188 consensus_queue.cc:237] T 4d4f47308d174a92bc73f23f2242be9a P 83f20e84af4c40d09a4804b6b26e697f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { replace: true } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { promote: true } }
I20240501 07:59:18.486672 8793 raft_consensus.cc:1261] T 4d4f47308d174a92bc73f23f2242be9a P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Refusing update from remote peer 83f20e84af4c40d09a4804b6b26e697f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:18.486688 9055 raft_consensus.cc:1261] T 4d4f47308d174a92bc73f23f2242be9a P a505e7cb133c4392bfda8a4a20874ae9 [term 1 FOLLOWER]: Refusing update from remote peer 83f20e84af4c40d09a4804b6b26e697f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 07:59:18.486788 9122 consensus_peers.cc:485] T 4d4f47308d174a92bc73f23f2242be9a P 83f20e84af4c40d09a4804b6b26e697f -> Peer 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129:46875): Couldn't send request to peer 8b8b3ad3aa8f4053b2167dd550f2086f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4d4f47308d174a92bc73f23f2242be9a. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:18.487282 13266 consensus_queue.cc:1035] T 4d4f47308d174a92bc73f23f2242be9a P 83f20e84af4c40d09a4804b6b26e697f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 07:59:18.487571 13266 consensus_queue.cc:1035] T 4d4f47308d174a92bc73f23f2242be9a P 83f20e84af4c40d09a4804b6b26e697f [LEADER]: Connected to new peer: Peer: permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 07:59:18.489337 13383 raft_consensus.cc:2932] T 4d4f47308d174a92bc73f23f2242be9a P 83f20e84af4c40d09a4804b6b26e697f [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { replace: true } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { promote: true } } }
I20240501 07:59:18.489778 8794 raft_consensus.cc:2932] T 4d4f47308d174a92bc73f23f2242be9a P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { replace: true } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { promote: true } } }
I20240501 07:59:18.492439 9055 raft_consensus.cc:2932] T 4d4f47308d174a92bc73f23f2242be9a P a505e7cb133c4392bfda8a4a20874ae9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { replace: true } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { promote: true } } }
I20240501 07:59:18.492492 6170 catalog_manager.cc:5484] T 4d4f47308d174a92bc73f23f2242be9a P 83f20e84af4c40d09a4804b6b26e697f reported cstate change: config changed from index -1 to 2, NON_VOTER 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129) added. New cstate: current_term: 1 leader_uuid: "83f20e84af4c40d09a4804b6b26e697f" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 07:59:18.502703 6577 consensus_queue.cc:237] T 144ff11d2a504f7ab7a230e6822d24ca P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { replace: true } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { promote: true } }
I20240501 07:59:18.504148 6311 raft_consensus.cc:1261] T 144ff11d2a504f7ab7a230e6822d24ca P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Refusing update from remote peer 3b164cbbc00f4e8eb3254ddd349a3ff9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:18.504179 9188 raft_consensus.cc:1261] T 144ff11d2a504f7ab7a230e6822d24ca P 83f20e84af4c40d09a4804b6b26e697f [term 1 FOLLOWER]: Refusing update from remote peer 3b164cbbc00f4e8eb3254ddd349a3ff9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 07:59:18.504380 6512 consensus_peers.cc:485] T 144ff11d2a504f7ab7a230e6822d24ca P 3b164cbbc00f4e8eb3254ddd349a3ff9 -> Peer 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130:32941): Couldn't send request to peer 8315d22c56f7439a9e1fc82f153f018c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 144ff11d2a504f7ab7a230e6822d24ca. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:18.504590 12119 consensus_queue.cc:1035] T 144ff11d2a504f7ab7a230e6822d24ca P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 07:59:18.504817 11107 consensus_queue.cc:1035] T 144ff11d2a504f7ab7a230e6822d24ca P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 07:59:18.506567 11107 raft_consensus.cc:2932] T 144ff11d2a504f7ab7a230e6822d24ca P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { replace: true } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { promote: true } } }
I20240501 07:59:18.507019 9188 raft_consensus.cc:2932] T 144ff11d2a504f7ab7a230e6822d24ca P 83f20e84af4c40d09a4804b6b26e697f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { replace: true } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { promote: true } } }
I20240501 07:59:18.507203 6311 raft_consensus.cc:2932] T 144ff11d2a504f7ab7a230e6822d24ca P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { replace: true } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { promote: true } } }
I20240501 07:59:18.510257 6168 catalog_manager.cc:5484] T 144ff11d2a504f7ab7a230e6822d24ca P 8b8b3ad3aa8f4053b2167dd550f2086f reported cstate change: config changed from index -1 to 2, NON_VOTER 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130) added. New cstate: current_term: 1 leader_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { replace: true } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { promote: true } } }
I20240501 07:59:18.513082 13351 raft_consensus.cc:482] T f0c1c05110894e39b41e6b5c5894b8d8 P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:18.513206 13351 raft_consensus.cc:504] T f0c1c05110894e39b41e6b5c5894b8d8 P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } }
I20240501 07:59:18.513644 13351 leader_election.cc:288] T f0c1c05110894e39b41e6b5c5894b8d8 P 13d05ebdca8c4f489c597ce550dd773d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149), 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133:39859)
I20240501 07:59:18.514034 8794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0c1c05110894e39b41e6b5c5894b8d8" candidate_uuid: "13d05ebdca8c4f489c597ce550dd773d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74077b61d6f54342bf9736b4a9f2eade" is_pre_election: true
I20240501 07:59:18.514189 8794 raft_consensus.cc:2454] T f0c1c05110894e39b41e6b5c5894b8d8 P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 13d05ebdca8c4f489c597ce550dd773d in term 0.
I20240501 07:59:18.514269 9188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0c1c05110894e39b41e6b5c5894b8d8" candidate_uuid: "13d05ebdca8c4f489c597ce550dd773d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83f20e84af4c40d09a4804b6b26e697f" is_pre_election: true
I20240501 07:59:18.514423 9188 raft_consensus.cc:2454] T f0c1c05110894e39b41e6b5c5894b8d8 P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 13d05ebdca8c4f489c597ce550dd773d in term 0.
I20240501 07:59:18.514565 8858 leader_election.cc:302] T f0c1c05110894e39b41e6b5c5894b8d8 P 13d05ebdca8c4f489c597ce550dd773d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13d05ebdca8c4f489c597ce550dd773d, 74077b61d6f54342bf9736b4a9f2eade; no voters:
I20240501 07:59:18.514758 13351 raft_consensus.cc:2781] T f0c1c05110894e39b41e6b5c5894b8d8 P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:18.514815 13351 raft_consensus.cc:482] T f0c1c05110894e39b41e6b5c5894b8d8 P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:18.514863 13351 raft_consensus.cc:3037] T f0c1c05110894e39b41e6b5c5894b8d8 P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:18.515954 13351 raft_consensus.cc:504] T f0c1c05110894e39b41e6b5c5894b8d8 P 13d05ebdca8c4f489c597ce550dd773d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } }
I20240501 07:59:18.516338 13351 leader_election.cc:288] T f0c1c05110894e39b41e6b5c5894b8d8 P 13d05ebdca8c4f489c597ce550dd773d [CANDIDATE]: Term 1 election: Requested vote from peers 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149), 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133:39859)
I20240501 07:59:18.516703 9188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0c1c05110894e39b41e6b5c5894b8d8" candidate_uuid: "13d05ebdca8c4f489c597ce550dd773d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83f20e84af4c40d09a4804b6b26e697f"
I20240501 07:59:18.516836 9188 raft_consensus.cc:3037] T f0c1c05110894e39b41e6b5c5894b8d8 P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:18.516829 8794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0c1c05110894e39b41e6b5c5894b8d8" candidate_uuid: "13d05ebdca8c4f489c597ce550dd773d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74077b61d6f54342bf9736b4a9f2eade"
I20240501 07:59:18.516953 8794 raft_consensus.cc:3037] T f0c1c05110894e39b41e6b5c5894b8d8 P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:18.517793 9188 raft_consensus.cc:2454] T f0c1c05110894e39b41e6b5c5894b8d8 P 83f20e84af4c40d09a4804b6b26e697f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13d05ebdca8c4f489c597ce550dd773d in term 1.
I20240501 07:59:18.518088 8859 leader_election.cc:302] T f0c1c05110894e39b41e6b5c5894b8d8 P 13d05ebdca8c4f489c597ce550dd773d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13d05ebdca8c4f489c597ce550dd773d, 83f20e84af4c40d09a4804b6b26e697f; no voters:
I20240501 07:59:18.518132 8794 raft_consensus.cc:2454] T f0c1c05110894e39b41e6b5c5894b8d8 P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13d05ebdca8c4f489c597ce550dd773d in term 1.
I20240501 07:59:18.518240 13351 raft_consensus.cc:2781] T f0c1c05110894e39b41e6b5c5894b8d8 P 13d05ebdca8c4f489c597ce550dd773d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:18.518342 13351 raft_consensus.cc:686] T f0c1c05110894e39b41e6b5c5894b8d8 P 13d05ebdca8c4f489c597ce550dd773d [term 1 LEADER]: Becoming Leader. State: Replica: 13d05ebdca8c4f489c597ce550dd773d, State: Running, Role: LEADER
I20240501 07:59:18.518479 13351 consensus_queue.cc:237] T f0c1c05110894e39b41e6b5c5894b8d8 P 13d05ebdca8c4f489c597ce550dd773d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } }
I20240501 07:59:18.520735 6170 catalog_manager.cc:5484] T f0c1c05110894e39b41e6b5c5894b8d8 P 13d05ebdca8c4f489c597ce550dd773d reported cstate change: term changed from 0 to 1, leader changed from <none> to 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134). New cstate: current_term: 1 leader_uuid: "13d05ebdca8c4f489c597ce550dd773d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } health_report { overall_health: HEALTHY } } }
I20240501 07:59:18.522339 8794 consensus_queue.cc:237] T 36aaf192042f4320b2526322424dc640 P 74077b61d6f54342bf9736b4a9f2eade [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { replace: true } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { promote: true } }
I20240501 07:59:18.523892 8925 raft_consensus.cc:1261] T 36aaf192042f4320b2526322424dc640 P 13d05ebdca8c4f489c597ce550dd773d [term 1 FOLLOWER]: Refusing update from remote peer 74077b61d6f54342bf9736b4a9f2eade: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 07:59:18.523911 8729 consensus_peers.cc:485] T 36aaf192042f4320b2526322424dc640 P 74077b61d6f54342bf9736b4a9f2eade -> Peer 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132:36773): Couldn't send request to peer 1f9cb72be98941b5a8e504cbe34132b9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 36aaf192042f4320b2526322424dc640. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:18.524045 9188 raft_consensus.cc:1261] T 36aaf192042f4320b2526322424dc640 P 83f20e84af4c40d09a4804b6b26e697f [term 1 FOLLOWER]: Refusing update from remote peer 74077b61d6f54342bf9736b4a9f2eade: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:18.524297 13396 consensus_queue.cc:1035] T 36aaf192042f4320b2526322424dc640 P 74077b61d6f54342bf9736b4a9f2eade [LEADER]: Connected to new peer: Peer: permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 07:59:18.524458 13218 consensus_queue.cc:1035] T 36aaf192042f4320b2526322424dc640 P 74077b61d6f54342bf9736b4a9f2eade [LEADER]: Connected to new peer: Peer: permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 07:59:18.526387 13538 raft_consensus.cc:2932] T 36aaf192042f4320b2526322424dc640 P 74077b61d6f54342bf9736b4a9f2eade [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { replace: true } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { promote: true } } }
I20240501 07:59:18.526687 8925 raft_consensus.cc:2932] T 36aaf192042f4320b2526322424dc640 P 13d05ebdca8c4f489c597ce550dd773d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { replace: true } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { promote: true } } }
I20240501 07:59:18.526803 9188 raft_consensus.cc:2932] T 36aaf192042f4320b2526322424dc640 P 83f20e84af4c40d09a4804b6b26e697f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { replace: true } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { promote: true } } }
I20240501 07:59:18.529702 6170 catalog_manager.cc:5484] T 36aaf192042f4320b2526322424dc640 P 13d05ebdca8c4f489c597ce550dd773d reported cstate change: config changed from index -1 to 2, NON_VOTER 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132) added. New cstate: current_term: 1 leader_uuid: "74077b61d6f54342bf9736b4a9f2eade" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { replace: true } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { promote: true } } }
I20240501 07:59:18.539238 9055 consensus_queue.cc:237] T 2560b1ec81ac4357b32c7a8ba9d72216 P a505e7cb133c4392bfda8a4a20874ae9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: true } }
I20240501 07:59:18.540670 6707 raft_consensus.cc:1261] T 2560b1ec81ac4357b32c7a8ba9d72216 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Refusing update from remote peer a505e7cb133c4392bfda8a4a20874ae9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 07:59:18.540906 8992 consensus_peers.cc:485] T 2560b1ec81ac4357b32c7a8ba9d72216 P a505e7cb133c4392bfda8a4a20874ae9 -> Peer 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131:34471): Couldn't send request to peer 3b164cbbc00f4e8eb3254ddd349a3ff9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2560b1ec81ac4357b32c7a8ba9d72216. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:18.540922 8794 raft_consensus.cc:1261] T 2560b1ec81ac4357b32c7a8ba9d72216 P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Refusing update from remote peer a505e7cb133c4392bfda8a4a20874ae9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:18.541249 11453 consensus_queue.cc:1035] T 2560b1ec81ac4357b32c7a8ba9d72216 P a505e7cb133c4392bfda8a4a20874ae9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 07:59:18.541443 13545 consensus_queue.cc:1035] T 2560b1ec81ac4357b32c7a8ba9d72216 P a505e7cb133c4392bfda8a4a20874ae9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 07:59:18.543381 13287 raft_consensus.cc:2932] T 2560b1ec81ac4357b32c7a8ba9d72216 P a505e7cb133c4392bfda8a4a20874ae9 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: true } } }
I20240501 07:59:18.543828 6707 raft_consensus.cc:2932] T 2560b1ec81ac4357b32c7a8ba9d72216 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: true } } }
I20240501 07:59:18.545127 8794 raft_consensus.cc:2932] T 2560b1ec81ac4357b32c7a8ba9d72216 P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: true } } }
I20240501 07:59:18.546991 6165 catalog_manager.cc:5484] T 2560b1ec81ac4357b32c7a8ba9d72216 P 1f9cb72be98941b5a8e504cbe34132b9 reported cstate change: config changed from index -1 to 2, NON_VOTER 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131) added. New cstate: current_term: 1 leader_uuid: "a505e7cb133c4392bfda8a4a20874ae9" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: true } } }
I20240501 07:59:18.556134 9188 consensus_queue.cc:237] T a93ce5841c7f40a9ab23f88efe7b85bb P 83f20e84af4c40d09a4804b6b26e697f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { replace: true } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { promote: true } }
I20240501 07:59:18.557720 8794 raft_consensus.cc:1261] T a93ce5841c7f40a9ab23f88efe7b85bb P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Refusing update from remote peer 83f20e84af4c40d09a4804b6b26e697f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:18.557855 9055 raft_consensus.cc:1261] T a93ce5841c7f40a9ab23f88efe7b85bb P a505e7cb133c4392bfda8a4a20874ae9 [term 1 FOLLOWER]: Refusing update from remote peer 83f20e84af4c40d09a4804b6b26e697f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:18.558303 13383 consensus_queue.cc:1035] T a93ce5841c7f40a9ab23f88efe7b85bb P 83f20e84af4c40d09a4804b6b26e697f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 07:59:18.558485 9122 consensus_peers.cc:485] T a93ce5841c7f40a9ab23f88efe7b85bb P 83f20e84af4c40d09a4804b6b26e697f -> Peer 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129:46875): Couldn't send request to peer 8b8b3ad3aa8f4053b2167dd550f2086f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a93ce5841c7f40a9ab23f88efe7b85bb. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:18.558488 13235 consensus_queue.cc:1035] T a93ce5841c7f40a9ab23f88efe7b85bb P 83f20e84af4c40d09a4804b6b26e697f [LEADER]: Connected to new peer: Peer: permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 07:59:18.560309 12649 raft_consensus.cc:2932] T a93ce5841c7f40a9ab23f88efe7b85bb P 83f20e84af4c40d09a4804b6b26e697f [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { replace: true } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { promote: true } } }
I20240501 07:59:18.560770 8793 raft_consensus.cc:2932] T a93ce5841c7f40a9ab23f88efe7b85bb P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { replace: true } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { promote: true } } }
I20240501 07:59:18.560777 9055 raft_consensus.cc:2932] T a93ce5841c7f40a9ab23f88efe7b85bb P a505e7cb133c4392bfda8a4a20874ae9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { replace: true } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { promote: true } } }
I20240501 07:59:18.561558 13545 raft_consensus.cc:482] T af718dc43add4edfac703013655699f4 P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:18.561645 13545 raft_consensus.cc:504] T af718dc43add4edfac703013655699f4 P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } }
I20240501 07:59:18.562000 13545 leader_election.cc:288] T af718dc43add4edfac703013655699f4 P a505e7cb133c4392bfda8a4a20874ae9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133:39859), 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131:34471)
I20240501 07:59:18.563812 6577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af718dc43add4edfac703013655699f4" candidate_uuid: "a505e7cb133c4392bfda8a4a20874ae9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" is_pre_election: true
I20240501 07:59:18.563952 6577 raft_consensus.cc:2454] T af718dc43add4edfac703013655699f4 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a505e7cb133c4392bfda8a4a20874ae9 in term 0.
I20240501 07:59:18.564500 8992 leader_election.cc:302] T af718dc43add4edfac703013655699f4 P a505e7cb133c4392bfda8a4a20874ae9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b164cbbc00f4e8eb3254ddd349a3ff9, a505e7cb133c4392bfda8a4a20874ae9; no voters:
I20240501 07:59:18.564375 6168 catalog_manager.cc:5484] T a93ce5841c7f40a9ab23f88efe7b85bb P 74077b61d6f54342bf9736b4a9f2eade reported cstate change: config changed from index -1 to 2, NON_VOTER 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129) added. New cstate: current_term: 1 leader_uuid: "83f20e84af4c40d09a4804b6b26e697f" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { replace: true } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { promote: true } } }
I20240501 07:59:18.564692 13287 raft_consensus.cc:2781] T af718dc43add4edfac703013655699f4 P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:18.564743 13287 raft_consensus.cc:482] T af718dc43add4edfac703013655699f4 P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:18.564779 13287 raft_consensus.cc:3037] T af718dc43add4edfac703013655699f4 P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:18.565775 13287 raft_consensus.cc:504] T af718dc43add4edfac703013655699f4 P a505e7cb133c4392bfda8a4a20874ae9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } }
I20240501 07:59:18.566110 13287 leader_election.cc:288] T af718dc43add4edfac703013655699f4 P a505e7cb133c4392bfda8a4a20874ae9 [CANDIDATE]: Term 1 election: Requested vote from peers 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133:39859), 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131:34471)
I20240501 07:59:18.566442 6575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af718dc43add4edfac703013655699f4" candidate_uuid: "a505e7cb133c4392bfda8a4a20874ae9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9"
I20240501 07:59:18.566586 6575 raft_consensus.cc:3037] T af718dc43add4edfac703013655699f4 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:18.567615 6575 raft_consensus.cc:2454] T af718dc43add4edfac703013655699f4 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a505e7cb133c4392bfda8a4a20874ae9 in term 1.
I20240501 07:59:18.568012 8992 leader_election.cc:302] T af718dc43add4edfac703013655699f4 P a505e7cb133c4392bfda8a4a20874ae9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b164cbbc00f4e8eb3254ddd349a3ff9, a505e7cb133c4392bfda8a4a20874ae9; no voters:
I20240501 07:59:18.568183 13287 raft_consensus.cc:2781] T af718dc43add4edfac703013655699f4 P a505e7cb133c4392bfda8a4a20874ae9 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:18.568259 13287 raft_consensus.cc:686] T af718dc43add4edfac703013655699f4 P a505e7cb133c4392bfda8a4a20874ae9 [term 1 LEADER]: Becoming Leader. State: Replica: a505e7cb133c4392bfda8a4a20874ae9, State: Running, Role: LEADER
I20240501 07:59:18.568262 8794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af718dc43add4edfac703013655699f4" candidate_uuid: "a505e7cb133c4392bfda8a4a20874ae9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74077b61d6f54342bf9736b4a9f2eade"
I20240501 07:59:18.568370 8794 raft_consensus.cc:3037] T af718dc43add4edfac703013655699f4 P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:18.568393 13287 consensus_queue.cc:237] T af718dc43add4edfac703013655699f4 P a505e7cb133c4392bfda8a4a20874ae9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } }
I20240501 07:59:18.569423 8794 raft_consensus.cc:2454] T af718dc43add4edfac703013655699f4 P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a505e7cb133c4392bfda8a4a20874ae9 in term 1.
I20240501 07:59:18.570307 8793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af718dc43add4edfac703013655699f4" candidate_uuid: "a505e7cb133c4392bfda8a4a20874ae9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74077b61d6f54342bf9736b4a9f2eade" is_pre_election: true
I20240501 07:59:18.570439 8793 raft_consensus.cc:2362] T af718dc43add4edfac703013655699f4 P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate a505e7cb133c4392bfda8a4a20874ae9 in term 1. Re-sending same reply.
I20240501 07:59:18.573514 6168 catalog_manager.cc:5484] T af718dc43add4edfac703013655699f4 P a505e7cb133c4392bfda8a4a20874ae9 reported cstate change: term changed from 0 to 1, leader changed from <none> to a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135). New cstate: current_term: 1 leader_uuid: "a505e7cb133c4392bfda8a4a20874ae9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } health_report { overall_health: UNKNOWN } } }
I20240501 07:59:18.575106 6577 consensus_queue.cc:237] T 0b24f67cf1a64debb23ee6618c820d33 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { replace: true } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: true } }
I20240501 07:59:18.576885 6444 raft_consensus.cc:1261] T 0b24f67cf1a64debb23ee6618c820d33 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Refusing update from remote peer 3b164cbbc00f4e8eb3254ddd349a3ff9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:18.576926 8793 raft_consensus.cc:1261] T 0b24f67cf1a64debb23ee6618c820d33 P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Refusing update from remote peer 3b164cbbc00f4e8eb3254ddd349a3ff9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 07:59:18.577034 6511 consensus_peers.cc:485] T 0b24f67cf1a64debb23ee6618c820d33 P 3b164cbbc00f4e8eb3254ddd349a3ff9 -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0b24f67cf1a64debb23ee6618c820d33. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:18.577339 13526 consensus_queue.cc:1035] T 0b24f67cf1a64debb23ee6618c820d33 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 07:59:18.577984 13526 consensus_queue.cc:1035] T 0b24f67cf1a64debb23ee6618c820d33 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 07:59:18.578629 6170 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55950:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 07:59:18.579521 13526 raft_consensus.cc:2932] T 0b24f67cf1a64debb23ee6618c820d33 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { replace: true } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: true } } }
I20240501 07:59:18.580374 8793 raft_consensus.cc:2932] T 0b24f67cf1a64debb23ee6618c820d33 P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { replace: true } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: true } } }
I20240501 07:59:18.582106 6444 raft_consensus.cc:2932] T 0b24f67cf1a64debb23ee6618c820d33 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { replace: true } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: true } } }
I20240501 07:59:18.584885 6165 catalog_manager.cc:5484] T 0b24f67cf1a64debb23ee6618c820d33 P 74077b61d6f54342bf9736b4a9f2eade reported cstate change: config changed from index -1 to 2, NON_VOTER 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { replace: true } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: true } } }
I20240501 07:59:18.585232 9188 raft_consensus.cc:1261] T 74d6caf4bbf345a0b7443780afd7fc01 P 83f20e84af4c40d09a4804b6b26e697f [term 1 FOLLOWER]: Refusing update from remote peer 74077b61d6f54342bf9736b4a9f2eade: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:18.585951 6577 raft_consensus.cc:1261] T 74d6caf4bbf345a0b7443780afd7fc01 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Refusing update from remote peer 74077b61d6f54342bf9736b4a9f2eade: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:18.586433 13396 consensus_queue.cc:1035] T 74d6caf4bbf345a0b7443780afd7fc01 P 74077b61d6f54342bf9736b4a9f2eade [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:18.586567 13218 consensus_queue.cc:1035] T 74d6caf4bbf345a0b7443780afd7fc01 P 74077b61d6f54342bf9736b4a9f2eade [LEADER]: Connected to new peer: Peer: permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:18.589598 6168 catalog_manager.cc:5576] TS 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149) does not have the latest schema for tablet 74d6caf4bbf345a0b7443780afd7fc01 (table rebalancer_dynamic_table [id=d23a3938211c43b4b7b7a3b54b1b6beb]). Expected version 1 got 0
I20240501 07:59:18.590876 6164 catalog_manager.cc:5576] TS 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131:34471) does not have the latest schema for tablet 74d6caf4bbf345a0b7443780afd7fc01 (table rebalancer_dynamic_table [id=d23a3938211c43b4b7b7a3b54b1b6beb]). Expected version 1 got 0
I20240501 07:59:18.591747 13395 tablet.cc:1722] T 74d6caf4bbf345a0b7443780afd7fc01 P 74077b61d6f54342bf9736b4a9f2eade: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 07:59:18.592049 13395 tablet.cc:1620] T 74d6caf4bbf345a0b7443780afd7fc01 P 74077b61d6f54342bf9736b4a9f2eade: MemRowSet was empty: no flush needed.
I20240501 07:59:18.593034 13092 tablet.cc:1722] T 74d6caf4bbf345a0b7443780afd7fc01 P 83f20e84af4c40d09a4804b6b26e697f: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 07:59:18.593353 13092 tablet.cc:1620] T 74d6caf4bbf345a0b7443780afd7fc01 P 83f20e84af4c40d09a4804b6b26e697f: MemRowSet was empty: no flush needed.
I20240501 07:59:18.595772 6168 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 0bc706e85ce74cef97a4827ea4d4aa65: rebalancer_dynamic_table [id=d23a3938211c43b4b7b7a3b54b1b6beb] alter complete (version 1)
I20240501 07:59:18.598513 13573 tablet.cc:1722] T 74d6caf4bbf345a0b7443780afd7fc01 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 07:59:18.598989 13573 tablet.cc:1620] T 74d6caf4bbf345a0b7443780afd7fc01 P 3b164cbbc00f4e8eb3254ddd349a3ff9: MemRowSet was empty: no flush needed.
I20240501 07:59:18.600391 6168 catalog_manager.cc:5576] TS 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131:34471) does not have the latest schema for tablet 74d6caf4bbf345a0b7443780afd7fc01 (table rebalancer_dynamic_table [id=d23a3938211c43b4b7b7a3b54b1b6beb]). Expected version 1 got 0
I20240501 07:59:18.601398 8924 consensus_queue.cc:237] T 0864c290fafa488abefa93d41c0a2efe P 13d05ebdca8c4f489c597ce550dd773d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { replace: true } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { promote: true } }
I20240501 07:59:18.602747 8793 raft_consensus.cc:1261] T 0864c290fafa488abefa93d41c0a2efe P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Refusing update from remote peer 13d05ebdca8c4f489c597ce550dd773d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 07:59:18.602926 8860 consensus_peers.cc:485] T 0864c290fafa488abefa93d41c0a2efe P 13d05ebdca8c4f489c597ce550dd773d -> Peer 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130:32941): Couldn't send request to peer 8315d22c56f7439a9e1fc82f153f018c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0864c290fafa488abefa93d41c0a2efe. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:18.603178 13418 consensus_queue.cc:1035] T 0864c290fafa488abefa93d41c0a2efe P 13d05ebdca8c4f489c597ce550dd773d [LEADER]: Connected to new peer: Peer: permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 07:59:18.603717 9188 raft_consensus.cc:1261] T 0864c290fafa488abefa93d41c0a2efe P 83f20e84af4c40d09a4804b6b26e697f [term 1 FOLLOWER]: Refusing update from remote peer 13d05ebdca8c4f489c597ce550dd773d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:18.604158 13418 consensus_queue.cc:1035] T 0864c290fafa488abefa93d41c0a2efe P 13d05ebdca8c4f489c597ce550dd773d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 07:59:18.605718 13351 raft_consensus.cc:2932] T 0864c290fafa488abefa93d41c0a2efe P 13d05ebdca8c4f489c597ce550dd773d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { replace: true } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { promote: true } } }
I20240501 07:59:18.606384 8793 raft_consensus.cc:2932] T 0864c290fafa488abefa93d41c0a2efe P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { replace: true } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { promote: true } } }
I20240501 07:59:18.607033 9188 raft_consensus.cc:2932] T 0864c290fafa488abefa93d41c0a2efe P 83f20e84af4c40d09a4804b6b26e697f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { replace: true } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { promote: true } } }
I20240501 07:59:18.608876 6165 catalog_manager.cc:5484] T 0864c290fafa488abefa93d41c0a2efe P 13d05ebdca8c4f489c597ce550dd773d reported cstate change: config changed from index -1 to 2, NON_VOTER 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130) added. New cstate: current_term: 1 leader_uuid: "13d05ebdca8c4f489c597ce550dd773d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 07:59:18.618570 6577 consensus_queue.cc:237] T d3f2250a62c6429e886753aaffcbe883 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { promote: true } }
I20240501 07:59:18.620023 9055 raft_consensus.cc:1261] T d3f2250a62c6429e886753aaffcbe883 P a505e7cb133c4392bfda8a4a20874ae9 [term 1 FOLLOWER]: Refusing update from remote peer 3b164cbbc00f4e8eb3254ddd349a3ff9: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 07:59:18.620011 9188 raft_consensus.cc:1261] T d3f2250a62c6429e886753aaffcbe883 P 83f20e84af4c40d09a4804b6b26e697f [term 1 FOLLOWER]: Refusing update from remote peer 3b164cbbc00f4e8eb3254ddd349a3ff9: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
W20240501 07:59:18.620191 6512 consensus_peers.cc:485] T d3f2250a62c6429e886753aaffcbe883 P 3b164cbbc00f4e8eb3254ddd349a3ff9 -> Peer 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132:36773): Couldn't send request to peer 1f9cb72be98941b5a8e504cbe34132b9. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:18.620532 13565 consensus_queue.cc:1035] T d3f2250a62c6429e886753aaffcbe883 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 07:59:18.620782 12119 consensus_queue.cc:1035] T d3f2250a62c6429e886753aaffcbe883 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 07:59:18.622640 13565 raft_consensus.cc:2932] T d3f2250a62c6429e886753aaffcbe883 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { promote: true } } }
I20240501 07:59:18.623266 9055 raft_consensus.cc:2932] T d3f2250a62c6429e886753aaffcbe883 P a505e7cb133c4392bfda8a4a20874ae9 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { promote: true } } }
I20240501 07:59:18.625355 9188 raft_consensus.cc:2932] T d3f2250a62c6429e886753aaffcbe883 P 83f20e84af4c40d09a4804b6b26e697f [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { promote: true } } }
I20240501 07:59:18.626077 6168 catalog_manager.cc:5484] T d3f2250a62c6429e886753aaffcbe883 P a505e7cb133c4392bfda8a4a20874ae9 reported cstate change: config changed from index 9 to 10, NON_VOTER 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132) added. New cstate: current_term: 1 leader_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { promote: true } } }
I20240501 07:59:18.636487 9188 consensus_queue.cc:237] T c86c420310ec42ffa5b24304fba74267 P 83f20e84af4c40d09a4804b6b26e697f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { replace: true } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: true } }
I20240501 07:59:18.637018 13172 raft_consensus.cc:482] T bf806039d8c742d8a80ad994829bdf18 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:18.637123 13172 raft_consensus.cc:504] T bf806039d8c742d8a80ad994829bdf18 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } }
I20240501 07:59:18.637925 13172 leader_election.cc:288] T bf806039d8c742d8a80ad994829bdf18 P 8315d22c56f7439a9e1fc82f153f018c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607), 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149)
I20240501 07:59:18.638383 9055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf806039d8c742d8a80ad994829bdf18" candidate_uuid: "8315d22c56f7439a9e1fc82f153f018c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a505e7cb133c4392bfda8a4a20874ae9" is_pre_election: true
W20240501 07:59:18.638527 9122 consensus_peers.cc:485] T c86c420310ec42ffa5b24304fba74267 P 83f20e84af4c40d09a4804b6b26e697f -> Peer 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133:39859): Couldn't send request to peer 74077b61d6f54342bf9736b4a9f2eade. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c86c420310ec42ffa5b24304fba74267. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:18.638532 9055 raft_consensus.cc:2454] T bf806039d8c742d8a80ad994829bdf18 P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8315d22c56f7439a9e1fc82f153f018c in term 0.
I20240501 07:59:18.638682 9189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf806039d8c742d8a80ad994829bdf18" candidate_uuid: "8315d22c56f7439a9e1fc82f153f018c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83f20e84af4c40d09a4804b6b26e697f" is_pre_election: true
I20240501 07:59:18.638767 9056 raft_consensus.cc:1261] T c86c420310ec42ffa5b24304fba74267 P a505e7cb133c4392bfda8a4a20874ae9 [term 1 FOLLOWER]: Refusing update from remote peer 83f20e84af4c40d09a4804b6b26e697f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:18.638793 9189 raft_consensus.cc:2454] T bf806039d8c742d8a80ad994829bdf18 P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8315d22c56f7439a9e1fc82f153f018c in term 0.
I20240501 07:59:18.638875 6380 leader_election.cc:302] T bf806039d8c742d8a80ad994829bdf18 P 8315d22c56f7439a9e1fc82f153f018c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8315d22c56f7439a9e1fc82f153f018c, a505e7cb133c4392bfda8a4a20874ae9; no voters:
I20240501 07:59:18.639102 13383 consensus_queue.cc:1035] T c86c420310ec42ffa5b24304fba74267 P 83f20e84af4c40d09a4804b6b26e697f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 07:59:18.639161 13172 raft_consensus.cc:2781] T bf806039d8c742d8a80ad994829bdf18 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:18.639212 13172 raft_consensus.cc:482] T bf806039d8c742d8a80ad994829bdf18 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:18.639271 13172 raft_consensus.cc:3037] T bf806039d8c742d8a80ad994829bdf18 P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:18.640506 13172 raft_consensus.cc:504] T bf806039d8c742d8a80ad994829bdf18 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } }
I20240501 07:59:18.640838 13172 leader_election.cc:288] T bf806039d8c742d8a80ad994829bdf18 P 8315d22c56f7439a9e1fc82f153f018c [CANDIDATE]: Term 1 election: Requested vote from peers a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607), 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149)
I20240501 07:59:18.641166 6444 raft_consensus.cc:1261] T c86c420310ec42ffa5b24304fba74267 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Refusing update from remote peer 83f20e84af4c40d09a4804b6b26e697f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:18.641278 9188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf806039d8c742d8a80ad994829bdf18" candidate_uuid: "8315d22c56f7439a9e1fc82f153f018c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83f20e84af4c40d09a4804b6b26e697f"
I20240501 07:59:18.641368 9188 raft_consensus.cc:3037] T bf806039d8c742d8a80ad994829bdf18 P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:18.641553 13383 consensus_queue.cc:1035] T c86c420310ec42ffa5b24304fba74267 P 83f20e84af4c40d09a4804b6b26e697f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 07:59:18.642159 9056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf806039d8c742d8a80ad994829bdf18" candidate_uuid: "8315d22c56f7439a9e1fc82f153f018c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a505e7cb133c4392bfda8a4a20874ae9"
I20240501 07:59:18.642269 9056 raft_consensus.cc:3037] T bf806039d8c742d8a80ad994829bdf18 P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:18.642268 9188 raft_consensus.cc:2454] T bf806039d8c742d8a80ad994829bdf18 P 83f20e84af4c40d09a4804b6b26e697f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8315d22c56f7439a9e1fc82f153f018c in term 1.
I20240501 07:59:18.642657 6380 leader_election.cc:302] T bf806039d8c742d8a80ad994829bdf18 P 8315d22c56f7439a9e1fc82f153f018c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8315d22c56f7439a9e1fc82f153f018c, 83f20e84af4c40d09a4804b6b26e697f; no voters:
I20240501 07:59:18.642946 13297 raft_consensus.cc:2781] T bf806039d8c742d8a80ad994829bdf18 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:18.643003 13297 raft_consensus.cc:686] T bf806039d8c742d8a80ad994829bdf18 P 8315d22c56f7439a9e1fc82f153f018c [term 1 LEADER]: Becoming Leader. State: Replica: 8315d22c56f7439a9e1fc82f153f018c, State: Running, Role: LEADER
I20240501 07:59:18.643113 13297 consensus_queue.cc:237] T bf806039d8c742d8a80ad994829bdf18 P 8315d22c56f7439a9e1fc82f153f018c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } }
I20240501 07:59:18.643301 9056 raft_consensus.cc:2454] T bf806039d8c742d8a80ad994829bdf18 P a505e7cb133c4392bfda8a4a20874ae9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8315d22c56f7439a9e1fc82f153f018c in term 1.
I20240501 07:59:18.645259 6164 catalog_manager.cc:5484] T bf806039d8c742d8a80ad994829bdf18 P 8315d22c56f7439a9e1fc82f153f018c reported cstate change: term changed from 0 to 1, leader changed from <none> to 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130). New cstate: current_term: 1 leader_uuid: "8315d22c56f7439a9e1fc82f153f018c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } health_report { overall_health: UNKNOWN } } }
I20240501 07:59:18.646956 13266 raft_consensus.cc:2932] T c86c420310ec42ffa5b24304fba74267 P 83f20e84af4c40d09a4804b6b26e697f [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { replace: true } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: true } } }
I20240501 07:59:18.648918 9056 raft_consensus.cc:2932] T c86c420310ec42ffa5b24304fba74267 P a505e7cb133c4392bfda8a4a20874ae9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { replace: true } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: true } } }
I20240501 07:59:18.649797 6444 raft_consensus.cc:2932] T c86c420310ec42ffa5b24304fba74267 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { replace: true } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: true } } }
I20240501 07:59:18.650041 6164 catalog_manager.cc:5484] T c86c420310ec42ffa5b24304fba74267 P 83f20e84af4c40d09a4804b6b26e697f reported cstate change: config changed from index -1 to 2, NON_VOTER 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "83f20e84af4c40d09a4804b6b26e697f" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 07:59:18.663401 9056 consensus_queue.cc:237] T c16b7962c18d4e02b850c7c7c678eae4 P a505e7cb133c4392bfda8a4a20874ae9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { promote: true } }
I20240501 07:59:18.664937 8924 raft_consensus.cc:1261] T c16b7962c18d4e02b850c7c7c678eae4 P 13d05ebdca8c4f489c597ce550dd773d [term 1 FOLLOWER]: Refusing update from remote peer a505e7cb133c4392bfda8a4a20874ae9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:18.665182 8793 raft_consensus.cc:1261] T c16b7962c18d4e02b850c7c7c678eae4 P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Refusing update from remote peer a505e7cb133c4392bfda8a4a20874ae9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 07:59:18.665349 8989 consensus_peers.cc:485] T c16b7962c18d4e02b850c7c7c678eae4 P a505e7cb133c4392bfda8a4a20874ae9 -> Peer 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129:46875): Couldn't send request to peer 8b8b3ad3aa8f4053b2167dd550f2086f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c16b7962c18d4e02b850c7c7c678eae4. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:18.665643 13287 consensus_queue.cc:1035] T c16b7962c18d4e02b850c7c7c678eae4 P a505e7cb133c4392bfda8a4a20874ae9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 07:59:18.665854 11453 consensus_queue.cc:1035] T c16b7962c18d4e02b850c7c7c678eae4 P a505e7cb133c4392bfda8a4a20874ae9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 07:59:18.667609 13287 raft_consensus.cc:2932] T c16b7962c18d4e02b850c7c7c678eae4 P a505e7cb133c4392bfda8a4a20874ae9 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { promote: true } } }
I20240501 07:59:18.668058 8924 raft_consensus.cc:2932] T c16b7962c18d4e02b850c7c7c678eae4 P 13d05ebdca8c4f489c597ce550dd773d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { promote: true } } }
I20240501 07:59:18.671056 6164 catalog_manager.cc:5484] T c16b7962c18d4e02b850c7c7c678eae4 P a505e7cb133c4392bfda8a4a20874ae9 reported cstate change: config changed from index -1 to 2, NON_VOTER 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129) added. New cstate: current_term: 1 leader_uuid: "a505e7cb133c4392bfda8a4a20874ae9" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 07:59:18.672454 8793 raft_consensus.cc:2932] T c16b7962c18d4e02b850c7c7c678eae4 P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { promote: true } } }
I20240501 07:59:18.681457 6707 consensus_queue.cc:237] T a29720541b7b4a79aaa810592ef7c4b1 P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { replace: true } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: true } }
I20240501 07:59:18.683213 9188 raft_consensus.cc:1261] T a29720541b7b4a79aaa810592ef7c4b1 P 83f20e84af4c40d09a4804b6b26e697f [term 1 FOLLOWER]: Refusing update from remote peer 1f9cb72be98941b5a8e504cbe34132b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 07:59:18.683364 6642 consensus_peers.cc:485] T a29720541b7b4a79aaa810592ef7c4b1 P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a29720541b7b4a79aaa810592ef7c4b1. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:18.683815 13608 consensus_queue.cc:1035] T a29720541b7b4a79aaa810592ef7c4b1 P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 07:59:18.684015 8924 raft_consensus.cc:1261] T a29720541b7b4a79aaa810592ef7c4b1 P 13d05ebdca8c4f489c597ce550dd773d [term 1 FOLLOWER]: Refusing update from remote peer 1f9cb72be98941b5a8e504cbe34132b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:18.684405 13608 consensus_queue.cc:1035] T a29720541b7b4a79aaa810592ef7c4b1 P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 07:59:18.685613 13495 raft_consensus.cc:2932] T a29720541b7b4a79aaa810592ef7c4b1 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { replace: true } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: true } } }
I20240501 07:59:18.687253 9188 raft_consensus.cc:2932] T a29720541b7b4a79aaa810592ef7c4b1 P 83f20e84af4c40d09a4804b6b26e697f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { replace: true } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: true } } }
I20240501 07:59:18.690853 6165 catalog_manager.cc:5484] T a29720541b7b4a79aaa810592ef7c4b1 P 83f20e84af4c40d09a4804b6b26e697f reported cstate change: config changed from index -1 to 2, NON_VOTER a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "1f9cb72be98941b5a8e504cbe34132b9" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { replace: true } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: true } } }
I20240501 07:59:18.696426 8924 raft_consensus.cc:2932] T a29720541b7b4a79aaa810592ef7c4b1 P 13d05ebdca8c4f489c597ce550dd773d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { replace: true } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: true } } }
I20240501 07:59:18.701066 6164 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55950:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 07:59:18.703686 8924 consensus_queue.cc:237] T e0583fa43994452ca7553cbf2309eba1 P 13d05ebdca8c4f489c597ce550dd773d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { replace: true } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { promote: true } }
I20240501 07:59:18.705840 9189 raft_consensus.cc:1261] T e0583fa43994452ca7553cbf2309eba1 P 83f20e84af4c40d09a4804b6b26e697f [term 1 FOLLOWER]: Refusing update from remote peer 13d05ebdca8c4f489c597ce550dd773d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:18.706642 13351 consensus_queue.cc:1035] T e0583fa43994452ca7553cbf2309eba1 P 13d05ebdca8c4f489c597ce550dd773d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 07:59:18.706991 9056 raft_consensus.cc:1261] T e0583fa43994452ca7553cbf2309eba1 P a505e7cb133c4392bfda8a4a20874ae9 [term 1 FOLLOWER]: Refusing update from remote peer 13d05ebdca8c4f489c597ce550dd773d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 07:59:18.707327 8860 consensus_peers.cc:485] T e0583fa43994452ca7553cbf2309eba1 P 13d05ebdca8c4f489c597ce550dd773d -> Peer 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130:32941): Couldn't send request to peer 8315d22c56f7439a9e1fc82f153f018c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e0583fa43994452ca7553cbf2309eba1. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:18.708197 13395 tablet.cc:1722] T 74d6caf4bbf345a0b7443780afd7fc01 P 74077b61d6f54342bf9736b4a9f2eade: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 07:59:18.708539 13395 tablet.cc:1620] T 74d6caf4bbf345a0b7443780afd7fc01 P 74077b61d6f54342bf9736b4a9f2eade: MemRowSet was empty: no flush needed.
I20240501 07:59:18.709347 13092 tablet.cc:1722] T 74d6caf4bbf345a0b7443780afd7fc01 P 83f20e84af4c40d09a4804b6b26e697f: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 07:59:18.709666 13092 tablet.cc:1620] T 74d6caf4bbf345a0b7443780afd7fc01 P 83f20e84af4c40d09a4804b6b26e697f: MemRowSet was empty: no flush needed.
I20240501 07:59:18.709770 13351 consensus_queue.cc:1035] T e0583fa43994452ca7553cbf2309eba1 P 13d05ebdca8c4f489c597ce550dd773d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 07:59:18.710471 13573 tablet.cc:1722] T 74d6caf4bbf345a0b7443780afd7fc01 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 07:59:18.710767 13573 tablet.cc:1620] T 74d6caf4bbf345a0b7443780afd7fc01 P 3b164cbbc00f4e8eb3254ddd349a3ff9: MemRowSet was empty: no flush needed.
I20240501 07:59:18.716595 13418 raft_consensus.cc:2932] T e0583fa43994452ca7553cbf2309eba1 P 13d05ebdca8c4f489c597ce550dd773d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { replace: true } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { promote: true } } }
I20240501 07:59:18.717013 6170 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 0bc706e85ce74cef97a4827ea4d4aa65: rebalancer_dynamic_table [id=d23a3938211c43b4b7b7a3b54b1b6beb] alter complete (version 2)
I20240501 07:59:18.717927 9055 raft_consensus.cc:2932] T e0583fa43994452ca7553cbf2309eba1 P a505e7cb133c4392bfda8a4a20874ae9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { replace: true } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { promote: true } } }
I20240501 07:59:18.719508 9188 raft_consensus.cc:2932] T e0583fa43994452ca7553cbf2309eba1 P 83f20e84af4c40d09a4804b6b26e697f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { replace: true } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { promote: true } } }
I20240501 07:59:18.720984 6168 catalog_manager.cc:5484] T e0583fa43994452ca7553cbf2309eba1 P 13d05ebdca8c4f489c597ce550dd773d reported cstate change: config changed from index -1 to 2, NON_VOTER 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130) added. New cstate: current_term: 1 leader_uuid: "13d05ebdca8c4f489c597ce550dd773d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 07:59:18.729897 9055 consensus_queue.cc:237] T 8f36eefcda3f475d9571e92fdeb52eb4 P a505e7cb133c4392bfda8a4a20874ae9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { replace: true } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: true } }
I20240501 07:59:18.731415 8924 raft_consensus.cc:1261] T 8f36eefcda3f475d9571e92fdeb52eb4 P 13d05ebdca8c4f489c597ce550dd773d [term 1 FOLLOWER]: Refusing update from remote peer a505e7cb133c4392bfda8a4a20874ae9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:18.731427 6444 raft_consensus.cc:1261] T 8f36eefcda3f475d9571e92fdeb52eb4 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Refusing update from remote peer a505e7cb133c4392bfda8a4a20874ae9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 07:59:18.731616 8992 consensus_peers.cc:485] T 8f36eefcda3f475d9571e92fdeb52eb4 P a505e7cb133c4392bfda8a4a20874ae9 -> Peer 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131:34471): Couldn't send request to peer 3b164cbbc00f4e8eb3254ddd349a3ff9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8f36eefcda3f475d9571e92fdeb52eb4. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:18.731864 13545 consensus_queue.cc:1035] T 8f36eefcda3f475d9571e92fdeb52eb4 P a505e7cb133c4392bfda8a4a20874ae9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 07:59:18.732062 11453 consensus_queue.cc:1035] T 8f36eefcda3f475d9571e92fdeb52eb4 P a505e7cb133c4392bfda8a4a20874ae9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 07:59:18.734025 13287 raft_consensus.cc:2932] T 8f36eefcda3f475d9571e92fdeb52eb4 P a505e7cb133c4392bfda8a4a20874ae9 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { replace: true } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: true } } }
I20240501 07:59:18.734483 8924 raft_consensus.cc:2932] T 8f36eefcda3f475d9571e92fdeb52eb4 P 13d05ebdca8c4f489c597ce550dd773d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { replace: true } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: true } } }
I20240501 07:59:18.734596 6444 raft_consensus.cc:2932] T 8f36eefcda3f475d9571e92fdeb52eb4 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { replace: true } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: true } } }
I20240501 07:59:18.737720 6165 catalog_manager.cc:5484] T 8f36eefcda3f475d9571e92fdeb52eb4 P 13d05ebdca8c4f489c597ce550dd773d reported cstate change: config changed from index -1 to 2, NON_VOTER 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131) added. New cstate: current_term: 1 leader_uuid: "a505e7cb133c4392bfda8a4a20874ae9" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { replace: true } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: true } } }
I20240501 07:59:18.746248 6311 consensus_queue.cc:237] T 1c32cd00bc344ae1b161d3f9b43b53d3 P 8b8b3ad3aa8f4053b2167dd550f2086f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { replace: true } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: true } }
I20240501 07:59:18.747682 6707 raft_consensus.cc:1261] T 1c32cd00bc344ae1b161d3f9b43b53d3 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Refusing update from remote peer 8b8b3ad3aa8f4053b2167dd550f2086f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:18.747705 9188 raft_consensus.cc:1261] T 1c32cd00bc344ae1b161d3f9b43b53d3 P 83f20e84af4c40d09a4804b6b26e697f [term 1 FOLLOWER]: Refusing update from remote peer 8b8b3ad3aa8f4053b2167dd550f2086f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:18.748111 13635 consensus_queue.cc:1035] T 1c32cd00bc344ae1b161d3f9b43b53d3 P 8b8b3ad3aa8f4053b2167dd550f2086f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 07:59:18.748312 6248 consensus_peers.cc:485] T 1c32cd00bc344ae1b161d3f9b43b53d3 P 8b8b3ad3aa8f4053b2167dd550f2086f -> Peer 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133:39859): Couldn't send request to peer 74077b61d6f54342bf9736b4a9f2eade. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1c32cd00bc344ae1b161d3f9b43b53d3. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:18.748328 13424 consensus_queue.cc:1035] T 1c32cd00bc344ae1b161d3f9b43b53d3 P 8b8b3ad3aa8f4053b2167dd550f2086f [LEADER]: Connected to new peer: Peer: permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 07:59:18.750432 13424 raft_consensus.cc:2932] T 1c32cd00bc344ae1b161d3f9b43b53d3 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { replace: true } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: true } } }
I20240501 07:59:18.750734 9189 raft_consensus.cc:2932] T 1c32cd00bc344ae1b161d3f9b43b53d3 P 83f20e84af4c40d09a4804b6b26e697f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { replace: true } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: true } } }
I20240501 07:59:18.750842 6707 raft_consensus.cc:2932] T 1c32cd00bc344ae1b161d3f9b43b53d3 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { replace: true } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: true } } }
I20240501 07:59:18.753944 6170 catalog_manager.cc:5484] T 1c32cd00bc344ae1b161d3f9b43b53d3 P 1f9cb72be98941b5a8e504cbe34132b9 reported cstate change: config changed from index -1 to 2, NON_VOTER 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { replace: true } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: true } } }
I20240501 07:59:18.762030 6311 consensus_queue.cc:237] T e5d564b9558946f585ca7a7a3b457a07 P 8b8b3ad3aa8f4053b2167dd550f2086f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { replace: true } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { promote: true } }
I20240501 07:59:18.763407 8793 raft_consensus.cc:1261] T e5d564b9558946f585ca7a7a3b457a07 P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Refusing update from remote peer 8b8b3ad3aa8f4053b2167dd550f2086f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 07:59:18.763454 6250 consensus_peers.cc:485] T e5d564b9558946f585ca7a7a3b457a07 P 8b8b3ad3aa8f4053b2167dd550f2086f -> Peer 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132:36773): Couldn't send request to peer 1f9cb72be98941b5a8e504cbe34132b9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e5d564b9558946f585ca7a7a3b457a07. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:18.763430 8924 raft_consensus.cc:1261] T e5d564b9558946f585ca7a7a3b457a07 P 13d05ebdca8c4f489c597ce550dd773d [term 1 FOLLOWER]: Refusing update from remote peer 8b8b3ad3aa8f4053b2167dd550f2086f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:18.763808 13424 consensus_queue.cc:1035] T e5d564b9558946f585ca7a7a3b457a07 P 8b8b3ad3aa8f4053b2167dd550f2086f [LEADER]: Connected to new peer: Peer: permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 07:59:18.763984 13635 consensus_queue.cc:1035] T e5d564b9558946f585ca7a7a3b457a07 P 8b8b3ad3aa8f4053b2167dd550f2086f [LEADER]: Connected to new peer: Peer: permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 07:59:18.765703 13634 raft_consensus.cc:2932] T e5d564b9558946f585ca7a7a3b457a07 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { replace: true } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { promote: true } } }
I20240501 07:59:18.766160 8924 raft_consensus.cc:2932] T e5d564b9558946f585ca7a7a3b457a07 P 13d05ebdca8c4f489c597ce550dd773d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { replace: true } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { promote: true } } }
I20240501 07:59:18.766155 8793 raft_consensus.cc:2932] T e5d564b9558946f585ca7a7a3b457a07 P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { replace: true } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { promote: true } } }
I20240501 07:59:18.769173 6168 catalog_manager.cc:5484] T e5d564b9558946f585ca7a7a3b457a07 P 13d05ebdca8c4f489c597ce550dd773d reported cstate change: config changed from index -1 to 2, NON_VOTER 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132) added. New cstate: current_term: 1 leader_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { replace: true } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { promote: true } } }
I20240501 07:59:18.779038 9055 consensus_queue.cc:237] T 12ce6a21d06c4ba889542758be9dc62d P a505e7cb133c4392bfda8a4a20874ae9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { replace: true } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: false } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { promote: true } }
I20240501 07:59:18.780627 6577 raft_consensus.cc:1261] T 12ce6a21d06c4ba889542758be9dc62d P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Refusing update from remote peer a505e7cb133c4392bfda8a4a20874ae9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 07:59:18.780683 8924 raft_consensus.cc:1261] T 12ce6a21d06c4ba889542758be9dc62d P 13d05ebdca8c4f489c597ce550dd773d [term 1 FOLLOWER]: Refusing update from remote peer a505e7cb133c4392bfda8a4a20874ae9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240501 07:59:18.780951 8991 consensus_peers.cc:485] T 12ce6a21d06c4ba889542758be9dc62d P a505e7cb133c4392bfda8a4a20874ae9 -> Peer 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132:36773): Couldn't send request to peer 1f9cb72be98941b5a8e504cbe34132b9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 12ce6a21d06c4ba889542758be9dc62d. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:18.781091 13545 consensus_queue.cc:1035] T 12ce6a21d06c4ba889542758be9dc62d P a505e7cb133c4392bfda8a4a20874ae9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:59:18.781250 13287 consensus_queue.cc:1035] T 12ce6a21d06c4ba889542758be9dc62d P a505e7cb133c4392bfda8a4a20874ae9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:59:18.783074 11453 raft_consensus.cc:2932] T 12ce6a21d06c4ba889542758be9dc62d P a505e7cb133c4392bfda8a4a20874ae9 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { replace: true } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: false } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { promote: true } } }
I20240501 07:59:18.783510 6577 raft_consensus.cc:2932] T 12ce6a21d06c4ba889542758be9dc62d P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { replace: true } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: false } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { promote: true } } }
I20240501 07:59:18.785676 8924 raft_consensus.cc:2932] T 12ce6a21d06c4ba889542758be9dc62d P 13d05ebdca8c4f489c597ce550dd773d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { replace: true } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: false } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { promote: true } } }
I20240501 07:59:18.786094 6170 catalog_manager.cc:5484] T 12ce6a21d06c4ba889542758be9dc62d P a505e7cb133c4392bfda8a4a20874ae9 reported cstate change: config changed from index 4 to 5, NON_VOTER 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132) added. New cstate: current_term: 1 leader_uuid: "a505e7cb133c4392bfda8a4a20874ae9" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 07:59:18.796231 8794 consensus_queue.cc:237] T 0f9e2549fe294a93b3b25081a15b2028 P 74077b61d6f54342bf9736b4a9f2eade [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { promote: true } }
I20240501 07:59:18.797677 6577 raft_consensus.cc:1261] T 0f9e2549fe294a93b3b25081a15b2028 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Refusing update from remote peer 74077b61d6f54342bf9736b4a9f2eade: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:18.797747 9055 raft_consensus.cc:1261] T 0f9e2549fe294a93b3b25081a15b2028 P a505e7cb133c4392bfda8a4a20874ae9 [term 1 FOLLOWER]: Refusing update from remote peer 74077b61d6f54342bf9736b4a9f2eade: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 07:59:18.798089 13617 consensus_queue.cc:1035] T 0f9e2549fe294a93b3b25081a15b2028 P 74077b61d6f54342bf9736b4a9f2eade [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 07:59:18.798457 13538 consensus_queue.cc:1035] T 0f9e2549fe294a93b3b25081a15b2028 P 74077b61d6f54342bf9736b4a9f2eade [LEADER]: Connected to new peer: Peer: permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 07:59:18.798995 8727 consensus_peers.cc:485] T 0f9e2549fe294a93b3b25081a15b2028 P 74077b61d6f54342bf9736b4a9f2eade -> Peer 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129:46875): Couldn't send request to peer 8b8b3ad3aa8f4053b2167dd550f2086f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0f9e2549fe294a93b3b25081a15b2028. This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:18.800001 13617 raft_consensus.cc:2932] T 0f9e2549fe294a93b3b25081a15b2028 P 74077b61d6f54342bf9736b4a9f2eade [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { promote: true } } }
I20240501 07:59:18.800473 6577 raft_consensus.cc:2932] T 0f9e2549fe294a93b3b25081a15b2028 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { promote: true } } }
I20240501 07:59:18.800819 9055 raft_consensus.cc:2932] T 0f9e2549fe294a93b3b25081a15b2028 P a505e7cb133c4392bfda8a4a20874ae9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { promote: true } } }
I20240501 07:59:18.803126 6164 catalog_manager.cc:5484] T 0f9e2549fe294a93b3b25081a15b2028 P 3b164cbbc00f4e8eb3254ddd349a3ff9 reported cstate change: config changed from index -1 to 2, NON_VOTER 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129) added. New cstate: current_term: 1 leader_uuid: "74077b61d6f54342bf9736b4a9f2eade" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { promote: true } } }
I20240501 07:59:18.822503 6170 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55950:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 07:59:18.828007 13395 tablet.cc:1722] T 74d6caf4bbf345a0b7443780afd7fc01 P 74077b61d6f54342bf9736b4a9f2eade: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 07:59:18.828594 13092 tablet.cc:1722] T 74d6caf4bbf345a0b7443780afd7fc01 P 83f20e84af4c40d09a4804b6b26e697f: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 07:59:18.828823 13573 tablet.cc:1722] T 74d6caf4bbf345a0b7443780afd7fc01 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 07:59:18.836038 6165 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 0bc706e85ce74cef97a4827ea4d4aa65: rebalancer_dynamic_table_new_name [id=d23a3938211c43b4b7b7a3b54b1b6beb] alter complete (version 3)
I20240501 07:59:18.939113 6170 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55950:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 07:59:18.939301 6170 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55950:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 07:59:18.941347 6170 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 0bc706e85ce74cef97a4827ea4d4aa65: Sending DeleteTablet for 3 replicas of tablet 74d6caf4bbf345a0b7443780afd7fc01
I20240501 07:59:18.942113 8774 tablet_service.cc:1514] Processing DeleteTablet for tablet 74d6caf4bbf345a0b7443780afd7fc01 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:18 UTC) from {username='slave'} at 127.0.0.1:43136
I20240501 07:59:18.942178 9167 tablet_service.cc:1514] Processing DeleteTablet for tablet 74d6caf4bbf345a0b7443780afd7fc01 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:18 UTC) from {username='slave'} at 127.0.0.1:33594
I20240501 07:59:18.942409 6556 tablet_service.cc:1514] Processing DeleteTablet for tablet 74d6caf4bbf345a0b7443780afd7fc01 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 07:59:18 UTC) from {username='slave'} at 127.0.0.1:53874
I20240501 07:59:18.942462 13747 tablet_replica.cc:330] T 74d6caf4bbf345a0b7443780afd7fc01 P 74077b61d6f54342bf9736b4a9f2eade: stopping tablet replica
I20240501 07:59:18.942646 13748 tablet_replica.cc:330] T 74d6caf4bbf345a0b7443780afd7fc01 P 83f20e84af4c40d09a4804b6b26e697f: stopping tablet replica
I20240501 07:59:18.942649 13747 raft_consensus.cc:2229] T 74d6caf4bbf345a0b7443780afd7fc01 P 74077b61d6f54342bf9736b4a9f2eade [term 1 LEADER]: Raft consensus shutting down.
I20240501 07:59:18.942752 13749 tablet_replica.cc:330] T 74d6caf4bbf345a0b7443780afd7fc01 P 3b164cbbc00f4e8eb3254ddd349a3ff9: stopping tablet replica
I20240501 07:59:18.942807 13748 raft_consensus.cc:2229] T 74d6caf4bbf345a0b7443780afd7fc01 P 83f20e84af4c40d09a4804b6b26e697f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:18.942878 13749 raft_consensus.cc:2229] T 74d6caf4bbf345a0b7443780afd7fc01 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 07:59:18.942888 13747 raft_consensus.cc:2258] T 74d6caf4bbf345a0b7443780afd7fc01 P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:18.942930 13748 raft_consensus.cc:2258] T 74d6caf4bbf345a0b7443780afd7fc01 P 83f20e84af4c40d09a4804b6b26e697f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:18.942986 13749 raft_consensus.cc:2258] T 74d6caf4bbf345a0b7443780afd7fc01 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 07:59:18.943567 13747 ts_tablet_manager.cc:1875] T 74d6caf4bbf345a0b7443780afd7fc01 P 74077b61d6f54342bf9736b4a9f2eade: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:59:18.943938 13748 ts_tablet_manager.cc:1875] T 74d6caf4bbf345a0b7443780afd7fc01 P 83f20e84af4c40d09a4804b6b26e697f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:59:18.945513 13749 ts_tablet_manager.cc:1875] T 74d6caf4bbf345a0b7443780afd7fc01 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 07:59:18.945906 13747 ts_tablet_manager.cc:1888] T 74d6caf4bbf345a0b7443780afd7fc01 P 74077b61d6f54342bf9736b4a9f2eade: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:59:18.945982 13747 log.cc:1196] T 74d6caf4bbf345a0b7443780afd7fc01 P 74077b61d6f54342bf9736b4a9f2eade: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/74d6caf4bbf345a0b7443780afd7fc01
I20240501 07:59:18.946079 13748 ts_tablet_manager.cc:1888] T 74d6caf4bbf345a0b7443780afd7fc01 P 83f20e84af4c40d09a4804b6b26e697f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:59:18.946161 13748 log.cc:1196] T 74d6caf4bbf345a0b7443780afd7fc01 P 83f20e84af4c40d09a4804b6b26e697f: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/74d6caf4bbf345a0b7443780afd7fc01
I20240501 07:59:18.946300 13747 ts_tablet_manager.cc:1909] T 74d6caf4bbf345a0b7443780afd7fc01 P 74077b61d6f54342bf9736b4a9f2eade: Deleting consensus metadata
I20240501 07:59:18.946437 13748 ts_tablet_manager.cc:1909] T 74d6caf4bbf345a0b7443780afd7fc01 P 83f20e84af4c40d09a4804b6b26e697f: Deleting consensus metadata
I20240501 07:59:18.947072 6157 catalog_manager.cc:4830] TS 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133:39859): tablet 74d6caf4bbf345a0b7443780afd7fc01 (table rebalancer_dynamic_table_new_name [id=d23a3938211c43b4b7b7a3b54b1b6beb]) successfully deleted
I20240501 07:59:18.947154 6158 catalog_manager.cc:4830] TS 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): tablet 74d6caf4bbf345a0b7443780afd7fc01 (table rebalancer_dynamic_table_new_name [id=d23a3938211c43b4b7b7a3b54b1b6beb]) successfully deleted
I20240501 07:59:18.948711 13749 ts_tablet_manager.cc:1888] T 74d6caf4bbf345a0b7443780afd7fc01 P 3b164cbbc00f4e8eb3254ddd349a3ff9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 07:59:18.948796 13749 log.cc:1196] T 74d6caf4bbf345a0b7443780afd7fc01 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Deleting WAL directory at /tmp/dist-test-task7tbgEt/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714550306730830-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/74d6caf4bbf345a0b7443780afd7fc01
I20240501 07:59:18.949180 13749 ts_tablet_manager.cc:1909] T 74d6caf4bbf345a0b7443780afd7fc01 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Deleting consensus metadata
I20240501 07:59:18.949944 6160 catalog_manager.cc:4830] TS 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131:34471): tablet 74d6caf4bbf345a0b7443780afd7fc01 (table rebalancer_dynamic_table_new_name [id=d23a3938211c43b4b7b7a3b54b1b6beb]) successfully deleted
I20240501 07:59:18.950934 13750 ts_tablet_manager.cc:902] T a93ce5841c7f40a9ab23f88efe7b85bb P 8b8b3ad3aa8f4053b2167dd550f2086f: Initiating tablet copy from peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149)
I20240501 07:59:18.951396 13750 tablet_copy_client.cc:308] T a93ce5841c7f40a9ab23f88efe7b85bb P 8b8b3ad3aa8f4053b2167dd550f2086f: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.136:46149
I20240501 07:59:18.951889 9207 tablet_copy_service.cc:140] P 83f20e84af4c40d09a4804b6b26e697f: Received BeginTabletCopySession request for tablet a93ce5841c7f40a9ab23f88efe7b85bb from peer 8b8b3ad3aa8f4053b2167dd550f2086f ({username='slave'} at 127.0.99.129:39491)
I20240501 07:59:18.951998 9207 tablet_copy_service.cc:161] P 83f20e84af4c40d09a4804b6b26e697f: Beginning new tablet copy session on tablet a93ce5841c7f40a9ab23f88efe7b85bb from peer 8b8b3ad3aa8f4053b2167dd550f2086f at {username='slave'} at 127.0.99.129:39491: session id = 8b8b3ad3aa8f4053b2167dd550f2086f-a93ce5841c7f40a9ab23f88efe7b85bb
I20240501 07:59:18.952893 9207 tablet_copy_source_session.cc:205] T a93ce5841c7f40a9ab23f88efe7b85bb P 83f20e84af4c40d09a4804b6b26e697f: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:18.953924 13750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a93ce5841c7f40a9ab23f88efe7b85bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:18.955842 13750 tablet_copy_client.cc:783] T a93ce5841c7f40a9ab23f88efe7b85bb P 8b8b3ad3aa8f4053b2167dd550f2086f: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:18.956007 13750 tablet_copy_client.cc:647] T a93ce5841c7f40a9ab23f88efe7b85bb P 8b8b3ad3aa8f4053b2167dd550f2086f: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:18.957196 13750 tablet_copy_client.cc:523] T a93ce5841c7f40a9ab23f88efe7b85bb P 8b8b3ad3aa8f4053b2167dd550f2086f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:18.958456 13750 tablet_bootstrap.cc:492] T a93ce5841c7f40a9ab23f88efe7b85bb P 8b8b3ad3aa8f4053b2167dd550f2086f: Bootstrap starting.
I20240501 07:59:18.961912 13750 tablet_bootstrap.cc:492] T a93ce5841c7f40a9ab23f88efe7b85bb P 8b8b3ad3aa8f4053b2167dd550f2086f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:18.962172 13750 tablet_bootstrap.cc:492] T a93ce5841c7f40a9ab23f88efe7b85bb P 8b8b3ad3aa8f4053b2167dd550f2086f: Bootstrap complete.
I20240501 07:59:18.962347 13750 ts_tablet_manager.cc:1367] T a93ce5841c7f40a9ab23f88efe7b85bb P 8b8b3ad3aa8f4053b2167dd550f2086f: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20240501 07:59:18.962721 13750 raft_consensus.cc:348] T a93ce5841c7f40a9ab23f88efe7b85bb P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { replace: true } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { promote: true } }
I20240501 07:59:18.962826 13750 raft_consensus.cc:729] T a93ce5841c7f40a9ab23f88efe7b85bb P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8b8b3ad3aa8f4053b2167dd550f2086f, State: Initialized, Role: LEARNER
I20240501 07:59:18.962909 13750 consensus_queue.cc:260] T a93ce5841c7f40a9ab23f88efe7b85bb P 8b8b3ad3aa8f4053b2167dd550f2086f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { replace: true } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { promote: true } }
I20240501 07:59:18.963181 13750 ts_tablet_manager.cc:1398] T a93ce5841c7f40a9ab23f88efe7b85bb P 8b8b3ad3aa8f4053b2167dd550f2086f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:59:18.963749 9207 tablet_copy_service.cc:342] P 83f20e84af4c40d09a4804b6b26e697f: Request end of tablet copy session 8b8b3ad3aa8f4053b2167dd550f2086f-a93ce5841c7f40a9ab23f88efe7b85bb received from {username='slave'} at 127.0.99.129:39491
I20240501 07:59:18.963835 9207 tablet_copy_service.cc:433] P 83f20e84af4c40d09a4804b6b26e697f: ending tablet copy session 8b8b3ad3aa8f4053b2167dd550f2086f-a93ce5841c7f40a9ab23f88efe7b85bb on tablet a93ce5841c7f40a9ab23f88efe7b85bb with peer 8b8b3ad3aa8f4053b2167dd550f2086f
I20240501 07:59:19.014304 13756 ts_tablet_manager.cc:902] T d3f2250a62c6429e886753aaffcbe883 P 1f9cb72be98941b5a8e504cbe34132b9: Initiating tablet copy from peer 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131:34471)
I20240501 07:59:19.014973 13756 tablet_copy_client.cc:308] T d3f2250a62c6429e886753aaffcbe883 P 1f9cb72be98941b5a8e504cbe34132b9: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:34471
I20240501 07:59:19.015506 6597 tablet_copy_service.cc:140] P 3b164cbbc00f4e8eb3254ddd349a3ff9: Received BeginTabletCopySession request for tablet d3f2250a62c6429e886753aaffcbe883 from peer 1f9cb72be98941b5a8e504cbe34132b9 ({username='slave'} at 127.0.99.132:33923)
I20240501 07:59:19.015621 6597 tablet_copy_service.cc:161] P 3b164cbbc00f4e8eb3254ddd349a3ff9: Beginning new tablet copy session on tablet d3f2250a62c6429e886753aaffcbe883 from peer 1f9cb72be98941b5a8e504cbe34132b9 at {username='slave'} at 127.0.99.132:33923: session id = 1f9cb72be98941b5a8e504cbe34132b9-d3f2250a62c6429e886753aaffcbe883
I20240501 07:59:19.016667 6597 tablet_copy_source_session.cc:205] T d3f2250a62c6429e886753aaffcbe883 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:19.017500 13756 ts_tablet_manager.cc:1875] T d3f2250a62c6429e886753aaffcbe883 P 1f9cb72be98941b5a8e504cbe34132b9: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240501 07:59:19.019626 13756 ts_tablet_manager.cc:1888] T d3f2250a62c6429e886753aaffcbe883 P 1f9cb72be98941b5a8e504cbe34132b9: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.8
I20240501 07:59:19.019932 13756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3f2250a62c6429e886753aaffcbe883. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:19.021138 13756 tablet_copy_client.cc:783] T d3f2250a62c6429e886753aaffcbe883 P 1f9cb72be98941b5a8e504cbe34132b9: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:19.021338 13756 tablet_copy_client.cc:647] T d3f2250a62c6429e886753aaffcbe883 P 1f9cb72be98941b5a8e504cbe34132b9: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:19.022511 13756 tablet_copy_client.cc:523] T d3f2250a62c6429e886753aaffcbe883 P 1f9cb72be98941b5a8e504cbe34132b9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:19.023540 13756 tablet_bootstrap.cc:492] T d3f2250a62c6429e886753aaffcbe883 P 1f9cb72be98941b5a8e504cbe34132b9: Bootstrap starting.
I20240501 07:59:19.030499 13762 ts_tablet_manager.cc:902] T 36aaf192042f4320b2526322424dc640 P 1f9cb72be98941b5a8e504cbe34132b9: Initiating tablet copy from peer 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133:39859)
I20240501 07:59:19.031064 13762 tablet_copy_client.cc:308] T 36aaf192042f4320b2526322424dc640 P 1f9cb72be98941b5a8e504cbe34132b9: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.133:39859
I20240501 07:59:19.031685 8814 tablet_copy_service.cc:140] P 74077b61d6f54342bf9736b4a9f2eade: Received BeginTabletCopySession request for tablet 36aaf192042f4320b2526322424dc640 from peer 1f9cb72be98941b5a8e504cbe34132b9 ({username='slave'} at 127.0.99.132:54045)
I20240501 07:59:19.031793 8814 tablet_copy_service.cc:161] P 74077b61d6f54342bf9736b4a9f2eade: Beginning new tablet copy session on tablet 36aaf192042f4320b2526322424dc640 from peer 1f9cb72be98941b5a8e504cbe34132b9 at {username='slave'} at 127.0.99.132:54045: session id = 1f9cb72be98941b5a8e504cbe34132b9-36aaf192042f4320b2526322424dc640
I20240501 07:59:19.032799 8814 tablet_copy_source_session.cc:205] T 36aaf192042f4320b2526322424dc640 P 74077b61d6f54342bf9736b4a9f2eade: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:19.033596 13767 ts_tablet_manager.cc:902] T c86c420310ec42ffa5b24304fba74267 P 74077b61d6f54342bf9736b4a9f2eade: Initiating tablet copy from peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149)
I20240501 07:59:19.033802 13762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36aaf192042f4320b2526322424dc640. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:19.034176 13767 tablet_copy_client.cc:308] T c86c420310ec42ffa5b24304fba74267 P 74077b61d6f54342bf9736b4a9f2eade: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.136:46149
I20240501 07:59:19.034617 9207 tablet_copy_service.cc:140] P 83f20e84af4c40d09a4804b6b26e697f: Received BeginTabletCopySession request for tablet c86c420310ec42ffa5b24304fba74267 from peer 74077b61d6f54342bf9736b4a9f2eade ({username='slave'} at 127.0.99.133:43505)
I20240501 07:59:19.034726 9207 tablet_copy_service.cc:161] P 83f20e84af4c40d09a4804b6b26e697f: Beginning new tablet copy session on tablet c86c420310ec42ffa5b24304fba74267 from peer 74077b61d6f54342bf9736b4a9f2eade at {username='slave'} at 127.0.99.133:43505: session id = 74077b61d6f54342bf9736b4a9f2eade-c86c420310ec42ffa5b24304fba74267
I20240501 07:59:19.035624 9207 tablet_copy_source_session.cc:205] T c86c420310ec42ffa5b24304fba74267 P 83f20e84af4c40d09a4804b6b26e697f: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:19.036128 13762 tablet_copy_client.cc:783] T 36aaf192042f4320b2526322424dc640 P 1f9cb72be98941b5a8e504cbe34132b9: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:19.036295 13762 tablet_copy_client.cc:647] T 36aaf192042f4320b2526322424dc640 P 1f9cb72be98941b5a8e504cbe34132b9: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:19.036648 13767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c86c420310ec42ffa5b24304fba74267. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:19.037287 13762 tablet_copy_client.cc:523] T 36aaf192042f4320b2526322424dc640 P 1f9cb72be98941b5a8e504cbe34132b9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:19.038404 13762 tablet_bootstrap.cc:492] T 36aaf192042f4320b2526322424dc640 P 1f9cb72be98941b5a8e504cbe34132b9: Bootstrap starting.
I20240501 07:59:19.038873 13767 tablet_copy_client.cc:783] T c86c420310ec42ffa5b24304fba74267 P 74077b61d6f54342bf9736b4a9f2eade: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:19.039017 13767 tablet_copy_client.cc:647] T c86c420310ec42ffa5b24304fba74267 P 74077b61d6f54342bf9736b4a9f2eade: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:19.040352 13767 tablet_copy_client.cc:523] T c86c420310ec42ffa5b24304fba74267 P 74077b61d6f54342bf9736b4a9f2eade: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:19.041652 13767 tablet_bootstrap.cc:492] T c86c420310ec42ffa5b24304fba74267 P 74077b61d6f54342bf9736b4a9f2eade: Bootstrap starting.
I20240501 07:59:19.042050 13762 tablet_bootstrap.cc:492] T 36aaf192042f4320b2526322424dc640 P 1f9cb72be98941b5a8e504cbe34132b9: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:19.042315 13762 tablet_bootstrap.cc:492] T 36aaf192042f4320b2526322424dc640 P 1f9cb72be98941b5a8e504cbe34132b9: Bootstrap complete.
I20240501 07:59:19.042459 13762 ts_tablet_manager.cc:1367] T 36aaf192042f4320b2526322424dc640 P 1f9cb72be98941b5a8e504cbe34132b9: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20240501 07:59:19.042943 13762 raft_consensus.cc:348] T 36aaf192042f4320b2526322424dc640 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { replace: true } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { promote: true } }
I20240501 07:59:19.043056 13762 raft_consensus.cc:729] T 36aaf192042f4320b2526322424dc640 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1f9cb72be98941b5a8e504cbe34132b9, State: Initialized, Role: LEARNER
I20240501 07:59:19.043313 13762 consensus_queue.cc:260] T 36aaf192042f4320b2526322424dc640 P 1f9cb72be98941b5a8e504cbe34132b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { replace: true } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { promote: true } }
I20240501 07:59:19.043603 6170 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55950:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 07:59:19.044363 13762 ts_tablet_manager.cc:1398] T 36aaf192042f4320b2526322424dc640 P 1f9cb72be98941b5a8e504cbe34132b9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 07:59:19.045720 8814 tablet_copy_service.cc:342] P 74077b61d6f54342bf9736b4a9f2eade: Request end of tablet copy session 1f9cb72be98941b5a8e504cbe34132b9-36aaf192042f4320b2526322424dc640 received from {username='slave'} at 127.0.99.132:54045
I20240501 07:59:19.045794 8814 tablet_copy_service.cc:433] P 74077b61d6f54342bf9736b4a9f2eade: ending tablet copy session 1f9cb72be98941b5a8e504cbe34132b9-36aaf192042f4320b2526322424dc640 on tablet 36aaf192042f4320b2526322424dc640 with peer 1f9cb72be98941b5a8e504cbe34132b9
I20240501 07:59:19.047380 13767 tablet_bootstrap.cc:492] T c86c420310ec42ffa5b24304fba74267 P 74077b61d6f54342bf9736b4a9f2eade: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:19.047748 13767 tablet_bootstrap.cc:492] T c86c420310ec42ffa5b24304fba74267 P 74077b61d6f54342bf9736b4a9f2eade: Bootstrap complete.
I20240501 07:59:19.047899 13767 ts_tablet_manager.cc:1367] T c86c420310ec42ffa5b24304fba74267 P 74077b61d6f54342bf9736b4a9f2eade: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.001s
I20240501 07:59:19.048800 13767 raft_consensus.cc:348] T c86c420310ec42ffa5b24304fba74267 P 74077b61d6f54342bf9736b4a9f2eade [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { replace: true } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: true } }
I20240501 07:59:19.048920 13767 raft_consensus.cc:729] T c86c420310ec42ffa5b24304fba74267 P 74077b61d6f54342bf9736b4a9f2eade [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 74077b61d6f54342bf9736b4a9f2eade, State: Initialized, Role: LEARNER
I20240501 07:59:19.049036 13767 consensus_queue.cc:260] T c86c420310ec42ffa5b24304fba74267 P 74077b61d6f54342bf9736b4a9f2eade [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { replace: true } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: true } }
I20240501 07:59:19.049290 13767 ts_tablet_manager.cc:1398] T c86c420310ec42ffa5b24304fba74267 P 74077b61d6f54342bf9736b4a9f2eade: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:19.050524 8905 tablet_service.cc:1467] Processing CreateTablet for tablet c197ba256dc74e6d92e34799c274b4e3 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=158b84b3e18943d38616a2fb6e606d35]), partition=
I20240501 07:59:19.050778 8905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c197ba256dc74e6d92e34799c274b4e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:19.051226 9207 tablet_copy_service.cc:342] P 83f20e84af4c40d09a4804b6b26e697f: Request end of tablet copy session 74077b61d6f54342bf9736b4a9f2eade-c86c420310ec42ffa5b24304fba74267 received from {username='slave'} at 127.0.99.133:43505
I20240501 07:59:19.051326 9207 tablet_copy_service.cc:433] P 83f20e84af4c40d09a4804b6b26e697f: ending tablet copy session 74077b61d6f54342bf9736b4a9f2eade-c86c420310ec42ffa5b24304fba74267 on tablet c86c420310ec42ffa5b24304fba74267 with peer 74077b61d6f54342bf9736b4a9f2eade
I20240501 07:59:19.053128 13774 tablet_bootstrap.cc:492] T c197ba256dc74e6d92e34799c274b4e3 P 13d05ebdca8c4f489c597ce550dd773d: Bootstrap starting.
I20240501 07:59:19.053203 9167 tablet_service.cc:1467] Processing CreateTablet for tablet c197ba256dc74e6d92e34799c274b4e3 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=158b84b3e18943d38616a2fb6e606d35]), partition=
I20240501 07:59:19.053439 9167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c197ba256dc74e6d92e34799c274b4e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:19.054427 13775 ts_tablet_manager.cc:902] T 144ff11d2a504f7ab7a230e6822d24ca P 8315d22c56f7439a9e1fc82f153f018c: Initiating tablet copy from peer 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131:34471)
I20240501 07:59:19.055009 13775 tablet_copy_client.cc:308] T 144ff11d2a504f7ab7a230e6822d24ca P 8315d22c56f7439a9e1fc82f153f018c: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:34471
I20240501 07:59:19.055693 13780 tablet_bootstrap.cc:492] T c197ba256dc74e6d92e34799c274b4e3 P 83f20e84af4c40d09a4804b6b26e697f: Bootstrap starting.
I20240501 07:59:19.055847 9036 tablet_service.cc:1467] Processing CreateTablet for tablet c197ba256dc74e6d92e34799c274b4e3 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=158b84b3e18943d38616a2fb6e606d35]), partition=
I20240501 07:59:19.055927 6597 tablet_copy_service.cc:140] P 3b164cbbc00f4e8eb3254ddd349a3ff9: Received BeginTabletCopySession request for tablet 144ff11d2a504f7ab7a230e6822d24ca from peer 8315d22c56f7439a9e1fc82f153f018c ({username='slave'} at 127.0.99.130:44571)
I20240501 07:59:19.056054 6597 tablet_copy_service.cc:161] P 3b164cbbc00f4e8eb3254ddd349a3ff9: Beginning new tablet copy session on tablet 144ff11d2a504f7ab7a230e6822d24ca from peer 8315d22c56f7439a9e1fc82f153f018c at {username='slave'} at 127.0.99.130:44571: session id = 8315d22c56f7439a9e1fc82f153f018c-144ff11d2a504f7ab7a230e6822d24ca
I20240501 07:59:19.056078 9036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c197ba256dc74e6d92e34799c274b4e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:19.056797 13780 tablet_bootstrap.cc:654] T c197ba256dc74e6d92e34799c274b4e3 P 83f20e84af4c40d09a4804b6b26e697f: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:19.057011 6597 tablet_copy_source_session.cc:205] T 144ff11d2a504f7ab7a230e6822d24ca P 3b164cbbc00f4e8eb3254ddd349a3ff9: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:19.055608 13774 tablet_bootstrap.cc:654] T c197ba256dc74e6d92e34799c274b4e3 P 13d05ebdca8c4f489c597ce550dd773d: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:19.057870 13775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 144ff11d2a504f7ab7a230e6822d24ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:19.059986 11453 consensus_queue.cc:1035] T af718dc43add4edfac703013655699f4 P a505e7cb133c4392bfda8a4a20874ae9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:19.060756 13775 tablet_copy_client.cc:783] T 144ff11d2a504f7ab7a230e6822d24ca P 8315d22c56f7439a9e1fc82f153f018c: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:19.060905 13775 tablet_copy_client.cc:647] T 144ff11d2a504f7ab7a230e6822d24ca P 8315d22c56f7439a9e1fc82f153f018c: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:19.062060 13780 tablet_bootstrap.cc:492] T c197ba256dc74e6d92e34799c274b4e3 P 83f20e84af4c40d09a4804b6b26e697f: No bootstrap required, opened a new log
I20240501 07:59:19.062351 13775 tablet_copy_client.cc:523] T 144ff11d2a504f7ab7a230e6822d24ca P 8315d22c56f7439a9e1fc82f153f018c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:19.063758 13775 tablet_bootstrap.cc:492] T 144ff11d2a504f7ab7a230e6822d24ca P 8315d22c56f7439a9e1fc82f153f018c: Bootstrap starting.
I20240501 07:59:19.065394 13780 ts_tablet_manager.cc:1367] T c197ba256dc74e6d92e34799c274b4e3 P 83f20e84af4c40d09a4804b6b26e697f: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.001s
I20240501 07:59:19.066226 13780 raft_consensus.cc:348] T c197ba256dc74e6d92e34799c274b4e3 P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } }
I20240501 07:59:19.066419 13780 raft_consensus.cc:374] T c197ba256dc74e6d92e34799c274b4e3 P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:19.066516 13780 raft_consensus.cc:729] T c197ba256dc74e6d92e34799c274b4e3 P 83f20e84af4c40d09a4804b6b26e697f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83f20e84af4c40d09a4804b6b26e697f, State: Initialized, Role: FOLLOWER
I20240501 07:59:19.067148 13780 consensus_queue.cc:260] T c197ba256dc74e6d92e34799c274b4e3 P 83f20e84af4c40d09a4804b6b26e697f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } }
I20240501 07:59:19.067799 13756 tablet_bootstrap.cc:492] T d3f2250a62c6429e886753aaffcbe883 P 1f9cb72be98941b5a8e504cbe34132b9: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:19.068080 13756 tablet_bootstrap.cc:492] T d3f2250a62c6429e886753aaffcbe883 P 1f9cb72be98941b5a8e504cbe34132b9: Bootstrap complete.
I20240501 07:59:19.068243 13756 ts_tablet_manager.cc:1367] T d3f2250a62c6429e886753aaffcbe883 P 1f9cb72be98941b5a8e504cbe34132b9: Time spent bootstrapping tablet: real 0.045s user 0.020s sys 0.004s
I20240501 07:59:19.068517 13774 tablet_bootstrap.cc:492] T c197ba256dc74e6d92e34799c274b4e3 P 13d05ebdca8c4f489c597ce550dd773d: No bootstrap required, opened a new log
I20240501 07:59:19.068660 13774 ts_tablet_manager.cc:1367] T c197ba256dc74e6d92e34799c274b4e3 P 13d05ebdca8c4f489c597ce550dd773d: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.000s
I20240501 07:59:19.068665 13780 ts_tablet_manager.cc:1398] T c197ba256dc74e6d92e34799c274b4e3 P 83f20e84af4c40d09a4804b6b26e697f: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240501 07:59:19.068658 13756 raft_consensus.cc:348] T d3f2250a62c6429e886753aaffcbe883 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { promote: true } }
I20240501 07:59:19.069278 13774 raft_consensus.cc:348] T c197ba256dc74e6d92e34799c274b4e3 P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } }
I20240501 07:59:19.069447 13774 raft_consensus.cc:374] T c197ba256dc74e6d92e34799c274b4e3 P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:19.071259 13774 raft_consensus.cc:729] T c197ba256dc74e6d92e34799c274b4e3 P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13d05ebdca8c4f489c597ce550dd773d, State: Initialized, Role: FOLLOWER
I20240501 07:59:19.071480 13774 consensus_queue.cc:260] T c197ba256dc74e6d92e34799c274b4e3 P 13d05ebdca8c4f489c597ce550dd773d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } }
I20240501 07:59:19.071877 13774 ts_tablet_manager.cc:1398] T c197ba256dc74e6d92e34799c274b4e3 P 13d05ebdca8c4f489c597ce550dd773d: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 07:59:19.073135 13781 tablet_bootstrap.cc:492] T c197ba256dc74e6d92e34799c274b4e3 P a505e7cb133c4392bfda8a4a20874ae9: Bootstrap starting.
I20240501 07:59:19.074225 13781 tablet_bootstrap.cc:654] T c197ba256dc74e6d92e34799c274b4e3 P a505e7cb133c4392bfda8a4a20874ae9: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:19.076635 13775 tablet_bootstrap.cc:492] T 144ff11d2a504f7ab7a230e6822d24ca P 8315d22c56f7439a9e1fc82f153f018c: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:19.076889 13775 tablet_bootstrap.cc:492] T 144ff11d2a504f7ab7a230e6822d24ca P 8315d22c56f7439a9e1fc82f153f018c: Bootstrap complete.
I20240501 07:59:19.077034 13756 raft_consensus.cc:729] T d3f2250a62c6429e886753aaffcbe883 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1f9cb72be98941b5a8e504cbe34132b9, State: Initialized, Role: LEARNER
I20240501 07:59:19.077098 13775 ts_tablet_manager.cc:1367] T 144ff11d2a504f7ab7a230e6822d24ca P 8315d22c56f7439a9e1fc82f153f018c: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.000s
I20240501 07:59:19.077185 13756 consensus_queue.cc:260] T d3f2250a62c6429e886753aaffcbe883 P 1f9cb72be98941b5a8e504cbe34132b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 1.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { promote: true } }
I20240501 07:59:19.077333 13545 consensus_queue.cc:1035] T af718dc43add4edfac703013655699f4 P a505e7cb133c4392bfda8a4a20874ae9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:19.077603 13756 ts_tablet_manager.cc:1398] T d3f2250a62c6429e886753aaffcbe883 P 1f9cb72be98941b5a8e504cbe34132b9: Time spent starting tablet: real 0.009s user 0.004s sys 0.000s
I20240501 07:59:19.077608 13775 raft_consensus.cc:348] T 144ff11d2a504f7ab7a230e6822d24ca P 8315d22c56f7439a9e1fc82f153f018c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { replace: true } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { promote: true } }
I20240501 07:59:19.077728 13775 raft_consensus.cc:729] T 144ff11d2a504f7ab7a230e6822d24ca P 8315d22c56f7439a9e1fc82f153f018c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8315d22c56f7439a9e1fc82f153f018c, State: Initialized, Role: LEARNER
I20240501 07:59:19.077826 13775 consensus_queue.cc:260] T 144ff11d2a504f7ab7a230e6822d24ca P 8315d22c56f7439a9e1fc82f153f018c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { replace: true } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { promote: true } }
I20240501 07:59:19.078106 13775 ts_tablet_manager.cc:1398] T 144ff11d2a504f7ab7a230e6822d24ca P 8315d22c56f7439a9e1fc82f153f018c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:19.078213 6597 tablet_copy_service.cc:342] P 3b164cbbc00f4e8eb3254ddd349a3ff9: Request end of tablet copy session 1f9cb72be98941b5a8e504cbe34132b9-d3f2250a62c6429e886753aaffcbe883 received from {username='slave'} at 127.0.99.132:33923
I20240501 07:59:19.078298 6597 tablet_copy_service.cc:433] P 3b164cbbc00f4e8eb3254ddd349a3ff9: ending tablet copy session 1f9cb72be98941b5a8e504cbe34132b9-d3f2250a62c6429e886753aaffcbe883 on tablet d3f2250a62c6429e886753aaffcbe883 with peer 1f9cb72be98941b5a8e504cbe34132b9
I20240501 07:59:19.078680 6596 tablet_copy_service.cc:342] P 3b164cbbc00f4e8eb3254ddd349a3ff9: Request end of tablet copy session 8315d22c56f7439a9e1fc82f153f018c-144ff11d2a504f7ab7a230e6822d24ca received from {username='slave'} at 127.0.99.130:44571
I20240501 07:59:19.078753 6596 tablet_copy_service.cc:433] P 3b164cbbc00f4e8eb3254ddd349a3ff9: ending tablet copy session 8315d22c56f7439a9e1fc82f153f018c-144ff11d2a504f7ab7a230e6822d24ca on tablet 144ff11d2a504f7ab7a230e6822d24ca with peer 8315d22c56f7439a9e1fc82f153f018c
I20240501 07:59:19.082369 13781 tablet_bootstrap.cc:492] T c197ba256dc74e6d92e34799c274b4e3 P a505e7cb133c4392bfda8a4a20874ae9: No bootstrap required, opened a new log
I20240501 07:59:19.082516 13781 ts_tablet_manager.cc:1367] T c197ba256dc74e6d92e34799c274b4e3 P a505e7cb133c4392bfda8a4a20874ae9: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240501 07:59:19.083132 13781 raft_consensus.cc:348] T c197ba256dc74e6d92e34799c274b4e3 P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } }
I20240501 07:59:19.083369 13781 raft_consensus.cc:374] T c197ba256dc74e6d92e34799c274b4e3 P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:19.083428 13781 raft_consensus.cc:729] T c197ba256dc74e6d92e34799c274b4e3 P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a505e7cb133c4392bfda8a4a20874ae9, State: Initialized, Role: FOLLOWER
I20240501 07:59:19.083566 13781 consensus_queue.cc:260] T c197ba256dc74e6d92e34799c274b4e3 P a505e7cb133c4392bfda8a4a20874ae9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } }
I20240501 07:59:19.083839 13781 ts_tablet_manager.cc:1398] T c197ba256dc74e6d92e34799c274b4e3 P a505e7cb133c4392bfda8a4a20874ae9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:19.092077 13789 ts_tablet_manager.cc:902] T 2560b1ec81ac4357b32c7a8ba9d72216 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Initiating tablet copy from peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607)
I20240501 07:59:19.092764 13789 tablet_copy_client.cc:308] T 2560b1ec81ac4357b32c7a8ba9d72216 P 3b164cbbc00f4e8eb3254ddd349a3ff9: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.135:42607
I20240501 07:59:19.093281 9076 tablet_copy_service.cc:140] P a505e7cb133c4392bfda8a4a20874ae9: Received BeginTabletCopySession request for tablet 2560b1ec81ac4357b32c7a8ba9d72216 from peer 3b164cbbc00f4e8eb3254ddd349a3ff9 ({username='slave'} at 127.0.99.131:39545)
I20240501 07:59:19.093389 9076 tablet_copy_service.cc:161] P a505e7cb133c4392bfda8a4a20874ae9: Beginning new tablet copy session on tablet 2560b1ec81ac4357b32c7a8ba9d72216 from peer 3b164cbbc00f4e8eb3254ddd349a3ff9 at {username='slave'} at 127.0.99.131:39545: session id = 3b164cbbc00f4e8eb3254ddd349a3ff9-2560b1ec81ac4357b32c7a8ba9d72216
I20240501 07:59:19.094381 9076 tablet_copy_source_session.cc:205] T 2560b1ec81ac4357b32c7a8ba9d72216 P a505e7cb133c4392bfda8a4a20874ae9: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:19.095526 13789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2560b1ec81ac4357b32c7a8ba9d72216. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:19.097843 13789 tablet_copy_client.cc:783] T 2560b1ec81ac4357b32c7a8ba9d72216 P 3b164cbbc00f4e8eb3254ddd349a3ff9: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:19.098496 13789 tablet_copy_client.cc:647] T 2560b1ec81ac4357b32c7a8ba9d72216 P 3b164cbbc00f4e8eb3254ddd349a3ff9: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:19.098757 13750 ts_tablet_manager.cc:902] T 4d4f47308d174a92bc73f23f2242be9a P 8b8b3ad3aa8f4053b2167dd550f2086f: Initiating tablet copy from peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149)
I20240501 07:59:19.099190 13750 tablet_copy_client.cc:308] T 4d4f47308d174a92bc73f23f2242be9a P 8b8b3ad3aa8f4053b2167dd550f2086f: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.136:46149
I20240501 07:59:19.099675 9207 tablet_copy_service.cc:140] P 83f20e84af4c40d09a4804b6b26e697f: Received BeginTabletCopySession request for tablet 4d4f47308d174a92bc73f23f2242be9a from peer 8b8b3ad3aa8f4053b2167dd550f2086f ({username='slave'} at 127.0.99.129:39491)
I20240501 07:59:19.099804 9207 tablet_copy_service.cc:161] P 83f20e84af4c40d09a4804b6b26e697f: Beginning new tablet copy session on tablet 4d4f47308d174a92bc73f23f2242be9a from peer 8b8b3ad3aa8f4053b2167dd550f2086f at {username='slave'} at 127.0.99.129:39491: session id = 8b8b3ad3aa8f4053b2167dd550f2086f-4d4f47308d174a92bc73f23f2242be9a
I20240501 07:59:19.100832 9207 tablet_copy_source_session.cc:205] T 4d4f47308d174a92bc73f23f2242be9a P 83f20e84af4c40d09a4804b6b26e697f: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:19.100885 13040 consensus_queue.cc:1035] T f0c1c05110894e39b41e6b5c5894b8d8 P 13d05ebdca8c4f489c597ce550dd773d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:19.101965 13040 consensus_queue.cc:1035] T f0c1c05110894e39b41e6b5c5894b8d8 P 13d05ebdca8c4f489c597ce550dd773d [LEADER]: Connected to new peer: Peer: permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:19.103222 13750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d4f47308d174a92bc73f23f2242be9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:19.105906 13750 tablet_copy_client.cc:783] T 4d4f47308d174a92bc73f23f2242be9a P 8b8b3ad3aa8f4053b2167dd550f2086f: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:19.106623 13750 tablet_copy_client.cc:647] T 4d4f47308d174a92bc73f23f2242be9a P 8b8b3ad3aa8f4053b2167dd550f2086f: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:19.107693 13750 tablet_copy_client.cc:523] T 4d4f47308d174a92bc73f23f2242be9a P 8b8b3ad3aa8f4053b2167dd550f2086f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:19.108151 13789 tablet_copy_client.cc:523] T 2560b1ec81ac4357b32c7a8ba9d72216 P 3b164cbbc00f4e8eb3254ddd349a3ff9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:19.109771 13789 tablet_bootstrap.cc:492] T 2560b1ec81ac4357b32c7a8ba9d72216 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Bootstrap starting.
I20240501 07:59:19.111980 13775 ts_tablet_manager.cc:902] T e0583fa43994452ca7553cbf2309eba1 P 8315d22c56f7439a9e1fc82f153f018c: Initiating tablet copy from peer 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134:40141)
I20240501 07:59:19.112457 13775 tablet_copy_client.cc:308] T e0583fa43994452ca7553cbf2309eba1 P 8315d22c56f7439a9e1fc82f153f018c: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.134:40141
I20240501 07:59:19.112534 13297 consensus_queue.cc:1035] T bf806039d8c742d8a80ad994829bdf18 P 8315d22c56f7439a9e1fc82f153f018c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:19.113072 8945 tablet_copy_service.cc:140] P 13d05ebdca8c4f489c597ce550dd773d: Received BeginTabletCopySession request for tablet e0583fa43994452ca7553cbf2309eba1 from peer 8315d22c56f7439a9e1fc82f153f018c ({username='slave'} at 127.0.99.130:51073)
I20240501 07:59:19.113169 8945 tablet_copy_service.cc:161] P 13d05ebdca8c4f489c597ce550dd773d: Beginning new tablet copy session on tablet e0583fa43994452ca7553cbf2309eba1 from peer 8315d22c56f7439a9e1fc82f153f018c at {username='slave'} at 127.0.99.130:51073: session id = 8315d22c56f7439a9e1fc82f153f018c-e0583fa43994452ca7553cbf2309eba1
I20240501 07:59:19.113514 13789 tablet_bootstrap.cc:492] T 2560b1ec81ac4357b32c7a8ba9d72216 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:19.113852 13789 tablet_bootstrap.cc:492] T 2560b1ec81ac4357b32c7a8ba9d72216 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Bootstrap complete.
I20240501 07:59:19.114030 13789 ts_tablet_manager.cc:1367] T 2560b1ec81ac4357b32c7a8ba9d72216 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.002s
I20240501 07:59:19.114135 8945 tablet_copy_source_session.cc:205] T e0583fa43994452ca7553cbf2309eba1 P 13d05ebdca8c4f489c597ce550dd773d: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:19.114451 13789 raft_consensus.cc:348] T 2560b1ec81ac4357b32c7a8ba9d72216 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: true } }
I20240501 07:59:19.114560 13789 raft_consensus.cc:729] T 2560b1ec81ac4357b32c7a8ba9d72216 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3b164cbbc00f4e8eb3254ddd349a3ff9, State: Initialized, Role: LEARNER
I20240501 07:59:19.114538 13750 tablet_bootstrap.cc:492] T 4d4f47308d174a92bc73f23f2242be9a P 8b8b3ad3aa8f4053b2167dd550f2086f: Bootstrap starting.
I20240501 07:59:19.114673 13789 consensus_queue.cc:260] T 2560b1ec81ac4357b32c7a8ba9d72216 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: true } }
I20240501 07:59:19.115051 13789 ts_tablet_manager.cc:1398] T 2560b1ec81ac4357b32c7a8ba9d72216 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:19.116196 9076 tablet_copy_service.cc:342] P a505e7cb133c4392bfda8a4a20874ae9: Request end of tablet copy session 3b164cbbc00f4e8eb3254ddd349a3ff9-2560b1ec81ac4357b32c7a8ba9d72216 received from {username='slave'} at 127.0.99.131:39545
I20240501 07:59:19.116288 9076 tablet_copy_service.cc:433] P a505e7cb133c4392bfda8a4a20874ae9: ending tablet copy session 3b164cbbc00f4e8eb3254ddd349a3ff9-2560b1ec81ac4357b32c7a8ba9d72216 on tablet 2560b1ec81ac4357b32c7a8ba9d72216 with peer 3b164cbbc00f4e8eb3254ddd349a3ff9
I20240501 07:59:19.117007 13775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0583fa43994452ca7553cbf2309eba1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:19.119092 13775 tablet_copy_client.cc:783] T e0583fa43994452ca7553cbf2309eba1 P 8315d22c56f7439a9e1fc82f153f018c: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:19.119227 13775 tablet_copy_client.cc:647] T e0583fa43994452ca7553cbf2309eba1 P 8315d22c56f7439a9e1fc82f153f018c: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:19.119527 13172 consensus_queue.cc:1035] T bf806039d8c742d8a80ad994829bdf18 P 8315d22c56f7439a9e1fc82f153f018c [LEADER]: Connected to new peer: Peer: permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:19.122165 13775 tablet_copy_client.cc:523] T e0583fa43994452ca7553cbf2309eba1 P 8315d22c56f7439a9e1fc82f153f018c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:19.122931 13750 tablet_bootstrap.cc:492] T 4d4f47308d174a92bc73f23f2242be9a P 8b8b3ad3aa8f4053b2167dd550f2086f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:19.123198 13750 tablet_bootstrap.cc:492] T 4d4f47308d174a92bc73f23f2242be9a P 8b8b3ad3aa8f4053b2167dd550f2086f: Bootstrap complete.
I20240501 07:59:19.123364 13750 ts_tablet_manager.cc:1367] T 4d4f47308d174a92bc73f23f2242be9a P 8b8b3ad3aa8f4053b2167dd550f2086f: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240501 07:59:19.123631 13775 tablet_bootstrap.cc:492] T e0583fa43994452ca7553cbf2309eba1 P 8315d22c56f7439a9e1fc82f153f018c: Bootstrap starting.
I20240501 07:59:19.123858 13750 raft_consensus.cc:348] T 4d4f47308d174a92bc73f23f2242be9a P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { replace: true } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { promote: true } }
I20240501 07:59:19.123980 13750 raft_consensus.cc:729] T 4d4f47308d174a92bc73f23f2242be9a P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8b8b3ad3aa8f4053b2167dd550f2086f, State: Initialized, Role: LEARNER
I20240501 07:59:19.124091 13750 consensus_queue.cc:260] T 4d4f47308d174a92bc73f23f2242be9a P 8b8b3ad3aa8f4053b2167dd550f2086f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { replace: true } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { promote: true } }
I20240501 07:59:19.124455 13750 ts_tablet_manager.cc:1398] T 4d4f47308d174a92bc73f23f2242be9a P 8b8b3ad3aa8f4053b2167dd550f2086f: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20240501 07:59:19.126204 9207 tablet_copy_service.cc:342] P 83f20e84af4c40d09a4804b6b26e697f: Request end of tablet copy session 8b8b3ad3aa8f4053b2167dd550f2086f-4d4f47308d174a92bc73f23f2242be9a received from {username='slave'} at 127.0.99.129:39491
I20240501 07:59:19.126282 9207 tablet_copy_service.cc:433] P 83f20e84af4c40d09a4804b6b26e697f: ending tablet copy session 8b8b3ad3aa8f4053b2167dd550f2086f-4d4f47308d174a92bc73f23f2242be9a on tablet 4d4f47308d174a92bc73f23f2242be9a with peer 8b8b3ad3aa8f4053b2167dd550f2086f
I20240501 07:59:19.128839 13811 ts_tablet_manager.cc:902] T 0b24f67cf1a64debb23ee6618c820d33 P 83f20e84af4c40d09a4804b6b26e697f: Initiating tablet copy from peer 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131:34471)
F20240501 07:59:19.129084 13811 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-0b24f67cf1a64debb23ee6618c820d33.queue_time_us, tablet-download-pool-0b24f67cf1a64debb23ee6618c820d33.queue_time_us) (tablet-download-pool-2e540f389b2a4aef93a020f549d11b67.queue_time_us, tablet-download-pool-2e540f389b2a4aef93a020f549d11b67.queue_time_us) (tablet-download-pool-2e540f389b2a4aef93a020f549d11b67.run_cpu_time_us, tablet-download-pool-2e540f389b2a4aef93a020f549d11b67.run_cpu_time_us) (tablet-download-pool-2e540f389b2a4aef93a020f549d11b67.run_wall_time_us, tablet-download-pool-2e540f389b2a4aef93a020f549d11b67.run_wall_time_us) (tablet-download-pool-404bb81b9c404dd288d2f8160a4273da.queue_time_us, tablet-download-pool-404bb81b9c404dd288d2f8160a4273da.queue_time_us) (tablet-download-pool-404bb81b9c404dd288d2f8160a4273da.run_cpu_time_us, tablet-download-pool-404bb81b9c404dd288d2f8160a4273da.run_cpu_time_us) (tablet-download-pool-404bb81b9c404dd288d2f8160a4273da.run_wall_time_us, tablet-download-pool-404bb81b9c404dd288d2f8160a4273da.run_wall_time_us) (tablet-download-pool-5ee738571af740c1b494d4f17413d1d8.queue_time_us, tablet-download-pool-5ee738571af740c1b494d4f17413d1d8.queue_time_us) (tablet-download-pool-5ee738571af740c1b494d4f17413d1d8.run_cpu_time_us, tablet-download-pool-5ee738571af740c1b494d4f17413d1d8.run_cpu_time_us) (tablet-download-pool-5ee738571af740c1b494d4f17413d1d8.run_wall_time_us, tablet-download-pool-5ee738571af740c1b494d4f17413d1d8.run_wall_time_us) (tablet-download-pool-6dc929e2a6ae45bd8b1f297583636bb6.queue_time_us, tablet-download-pool-6dc929e2a6ae45bd8b1f297583636bb6.queue_time_us) (tablet-download-pool-6dc929e2a6ae45bd8b1f297583636bb6.run_cpu_time_us, tablet-download-pool-6dc929e2a6ae45bd8b1f297583636bb6.run_cpu_time_us) (tablet-download-pool-6dc929e2a6ae45bd8b1f297583636bb6.run_wall_time_us, tablet-download-pool-6dc929e2a6ae45bd8b1f297583636bb6.run_wall_time_us) (tablet-download-pool-6e2ac8c8857d45b88d3b51147d2609aa.queue_time_us, tablet-download-pool-6e2ac8c8857d45b88d3b51147d2609aa.queue_time_us) (tablet-download-pool-6e2ac8c8857d45b88d3b51147d2609aa.run_cpu_time_us, tablet-download-pool-6e2ac8c8857d45b88d3b51147d2609aa.run_cpu_time_us) (tablet-download-pool-6e2ac8c8857d45b88d3b51147d2609aa.run_wall_time_us, tablet-download-pool-6e2ac8c8857d45b88d3b51147d2609aa.run_wall_time_us) (tablet-download-pool-782e37a4ebb34a68b461a74115ae8703.queue_time_us, tablet-download-pool-782e37a4ebb34a68b461a74115ae8703.queue_time_us) (tablet-download-pool-782e37a4ebb34a68b461a74115ae8703.run_cpu_time_us, tablet-download-pool-782e37a4ebb34a68b461a74115ae8703.run_cpu_time_us) (tablet-download-pool-782e37a4ebb34a68b461a74115ae8703.run_wall_time_us, tablet-download-pool-782e37a4ebb34a68b461a74115ae8703.run_wall_time_us) (tablet-download-pool-7d51b1940b594c79a7d13d7026b76da8.queue_time_us, tablet-download-pool-7d51b1940b594c79a7d13d7026b76da8.queue_time_us) (tablet-download-pool-7d51b1940b594c79a7d13d7026b76da8.run_cpu_time_us, tablet-download-pool-7d51b1940b594c79a7d13d7026b76da8.run_cpu_time_us) (tablet-download-pool-7d51b1940b594c79a7d13d7026b76da8.run_wall_time_us, tablet-download-pool-7d51b1940b594c79a7d13d7026b76da8.run_wall_time_us) (tablet-download-pool-aa1b8716228b4fd9af95870b7a613fb2.queue_time_us, tablet-download-pool-aa1b8716228b4fd9af95870b7a613fb2.queue_time_us) (tablet-download-pool-aa1b8716228b4fd9af95870b7a613fb2.run_cpu_time_us, tablet-download-pool-aa1b8716228b4fd9af95870b7a613fb2.run_cpu_time_us) (tablet-download-pool-aa1b8716228b4fd9af95870b7a613fb2.run_wall_time_us, tablet-download-pool-aa1b8716228b4fd9af95870b7a613fb2.run_wall_time_us) (tablet-download-pool-b7d57b7fd5234f50afbc859bc997f3e1.queue_time_us, tablet-download-pool-b7d57b7fd5234f50afbc859bc997f3e1.queue_time_us) (tablet-download-pool-b7d57b7fd5234f50afbc859bc997f3e1.run_cpu_time_us, tablet-download-pool-b7d57b7fd5234f50afbc859bc997f3e1.run_cpu_time_us) (tablet-download-pool-b7d57b7fd5234f50afbc859bc997f3e1.run_wall_time_us, tablet-download-pool-b7d57b7fd5234f50afbc859bc997f3e1.run_wall_time_us) (tablet-download-pool-c3a9036ccfee45498bca9681518d3699.queue_time_us, tablet-download-pool-c3a9036ccfee45498bca9681518d3699.queue_time_us) (tablet-download-pool-c3a9036ccfee45498bca9681518d3699.run_cpu_time_us, tablet-download-pool-c3a9036ccfee45498bca9681518d3699.run_cpu_time_us) (tablet-download-pool-c3a9036ccfee45498bca9681518d3699.run_wall_time_us, tablet-download-pool-c3a9036ccfee45498bca9681518d3699.run_wall_time_us) (tablet-download-pool-d3f2250a62c6429e886753aaffcbe883.queue_time_us, tablet-download-pool-d3f2250a62c6429e886753aaffcbe883.queue_time_us) (tablet-download-pool-d3f2250a62c6429e886753aaffcbe883.run_cpu_time_us, tablet-download-pool-d3f2250a62c6429e886753aaffcbe883.run_cpu_time_us) (tablet-download-pool-d3f2250a62c6429e886753aaffcbe883.run_wall_time_us, tablet-download-pool-d3f2250a62c6429e886753aaffcbe883.run_wall_time_us) (tablet-download-pool-f68dbc1f65eb4f1b91a62e2de82a7676.queue_time_us, tablet-download-pool-f68dbc1f65eb4f1b91a62e2de82a7676.queue_time_us) (tablet-download-pool-f68dbc1f65eb4f1b91a62e2de82a7676.run_cpu_time_us, tablet-download-pool-f68dbc1f65eb4f1b91a62e2de82a7676.run_cpu_time_us) (tablet-download-pool-f68dbc1f65eb4f1b91a62e2de82a7676.run_wall_time_us, tablet-download-pool-f68dbc1f65eb4f1b91a62e2de82a7676.run_wall_time_us) (tablet-download-pool-f9df6f3df2b94e5eb97dbff04dbe2ae1.queue_time_us, tablet-download-pool-f9df6f3df2b94e5eb97dbff04dbe2ae1.queue_time_us) (tablet-download-pool-f9df6f3df2b94e5eb97dbff04dbe2ae1.run_cpu_time_us, tablet-download-pool-f9df6f3df2b94e5eb97dbff04dbe2ae1.run_cpu_time_us) (tablet-download-pool-f9df6f3df2b94e5eb97dbff04dbe2ae1.run_wall_time_us, tablet-download-pool-f9df6f3df2b94e5eb97dbff04dbe2ae1.run_wall_time_us) (tablet-download-pool-ff9dc384dbac47dc9795617c64688943.queue_time_us, tablet-download-pool-ff9dc384dbac47dc9795617c64688943.queue_time_us) (tablet-download-pool-ff9dc384dbac47dc9795617c64688943.run_cpu_time_us, tablet-download-pool-ff9dc384dbac47dc9795617c64688943.run_cpu_time_us) (tablet-download-pool-ff9dc384dbac47dc9795617c64688943.run_wall_time_us, tablet-download-pool-ff9dc384dbac47dc9795617c64688943.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
I20240501 07:59:19.129328 13767 ts_tablet_manager.cc:902] T 1c32cd00bc344ae1b161d3f9b43b53d3 P 74077b61d6f54342bf9736b4a9f2eade: Initiating tablet copy from peer 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129:46875)
*** Check failure stack trace: ***
I20240501 07:59:19.129798 13767 tablet_copy_client.cc:308] T 1c32cd00bc344ae1b161d3f9b43b53d3 P 74077b61d6f54342bf9736b4a9f2eade: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:46875
I20240501 07:59:19.130219 6335 tablet_copy_service.cc:140] P 8b8b3ad3aa8f4053b2167dd550f2086f: Received BeginTabletCopySession request for tablet 1c32cd00bc344ae1b161d3f9b43b53d3 from peer 74077b61d6f54342bf9736b4a9f2eade ({username='slave'} at 127.0.99.133:50111)
I20240501 07:59:19.130312 6335 tablet_copy_service.cc:161] P 8b8b3ad3aa8f4053b2167dd550f2086f: Beginning new tablet copy session on tablet 1c32cd00bc344ae1b161d3f9b43b53d3 from peer 74077b61d6f54342bf9736b4a9f2eade at {username='slave'} at 127.0.99.133:50111: session id = 74077b61d6f54342bf9736b4a9f2eade-1c32cd00bc344ae1b161d3f9b43b53d3
I20240501 07:59:19.130597 13775 tablet_bootstrap.cc:492] T e0583fa43994452ca7553cbf2309eba1 P 8315d22c56f7439a9e1fc82f153f018c: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:19.130882 13775 tablet_bootstrap.cc:492] T e0583fa43994452ca7553cbf2309eba1 P 8315d22c56f7439a9e1fc82f153f018c: Bootstrap complete.
I20240501 07:59:19.131052 13775 ts_tablet_manager.cc:1367] T e0583fa43994452ca7553cbf2309eba1 P 8315d22c56f7439a9e1fc82f153f018c: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20240501 07:59:19.131292 6335 tablet_copy_source_session.cc:205] T 1c32cd00bc344ae1b161d3f9b43b53d3 P 8b8b3ad3aa8f4053b2167dd550f2086f: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:19.131645 13775 raft_consensus.cc:348] T e0583fa43994452ca7553cbf2309eba1 P 8315d22c56f7439a9e1fc82f153f018c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { replace: true } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { promote: true } }
I20240501 07:59:19.131778 13775 raft_consensus.cc:729] T e0583fa43994452ca7553cbf2309eba1 P 8315d22c56f7439a9e1fc82f153f018c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8315d22c56f7439a9e1fc82f153f018c, State: Initialized, Role: LEARNER
I20240501 07:59:19.131901 13775 consensus_queue.cc:260] T e0583fa43994452ca7553cbf2309eba1 P 8315d22c56f7439a9e1fc82f153f018c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { replace: true } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { promote: true } }
@ 0x7fb0b9804dcd google::LogMessage::Fail() at ??:0
I20240501 07:59:19.133700 13767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c32cd00bc344ae1b161d3f9b43b53d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:19.134282 13775 ts_tablet_manager.cc:1398] T e0583fa43994452ca7553cbf2309eba1 P 8315d22c56f7439a9e1fc82f153f018c: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20240501 07:59:19.134842 8945 tablet_copy_service.cc:342] P 13d05ebdca8c4f489c597ce550dd773d: Request end of tablet copy session 8315d22c56f7439a9e1fc82f153f018c-e0583fa43994452ca7553cbf2309eba1 received from {username='slave'} at 127.0.99.130:51073
I20240501 07:59:19.134928 8945 tablet_copy_service.cc:433] P 13d05ebdca8c4f489c597ce550dd773d: ending tablet copy session 8315d22c56f7439a9e1fc82f153f018c-e0583fa43994452ca7553cbf2309eba1 on tablet e0583fa43994452ca7553cbf2309eba1 with peer 8315d22c56f7439a9e1fc82f153f018c
I20240501 07:59:19.135977 13767 tablet_copy_client.cc:783] T 1c32cd00bc344ae1b161d3f9b43b53d3 P 74077b61d6f54342bf9736b4a9f2eade: tablet copy: Starting download of 0 data blocks...
@ 0x7fb0b9808b93 google::LogMessage::SendToLog() at ??:0
I20240501 07:59:19.136134 13767 tablet_copy_client.cc:647] T 1c32cd00bc344ae1b161d3f9b43b53d3 P 74077b61d6f54342bf9736b4a9f2eade: tablet copy: Starting download of 1 WAL segments...
@ 0x7fb0b98047cc google::LogMessage::Flush() at ??:0
I20240501 07:59:19.139708 13767 tablet_copy_client.cc:523] T 1c32cd00bc344ae1b161d3f9b43b53d3 P 74077b61d6f54342bf9736b4a9f2eade: tablet copy: Tablet Copy complete. Replacing tablet superblock.
@ 0x7fb0b9805f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
I20240501 07:59:19.141762 13767 tablet_bootstrap.cc:492] T 1c32cd00bc344ae1b161d3f9b43b53d3 P 74077b61d6f54342bf9736b4a9f2eade: Bootstrap starting.
@ 0x7fb0bac6e044 kudu::TraceMetrics::InternName() at ??:0
@ 0x7fb0bac52f98 kudu::ThreadPool::ThreadPool() at ??:0
I20240501 07:59:19.149667 13767 tablet_bootstrap.cc:492] T 1c32cd00bc344ae1b161d3f9b43b53d3 P 74077b61d6f54342bf9736b4a9f2eade: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:19.149950 13767 tablet_bootstrap.cc:492] T 1c32cd00bc344ae1b161d3f9b43b53d3 P 74077b61d6f54342bf9736b4a9f2eade: Bootstrap complete.
I20240501 07:59:19.150126 13767 ts_tablet_manager.cc:1367] T 1c32cd00bc344ae1b161d3f9b43b53d3 P 74077b61d6f54342bf9736b4a9f2eade: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.005s
I20240501 07:59:19.150693 13767 raft_consensus.cc:348] T 1c32cd00bc344ae1b161d3f9b43b53d3 P 74077b61d6f54342bf9736b4a9f2eade [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { replace: true } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: true } }
I20240501 07:59:19.150846 13767 raft_consensus.cc:729] T 1c32cd00bc344ae1b161d3f9b43b53d3 P 74077b61d6f54342bf9736b4a9f2eade [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 74077b61d6f54342bf9736b4a9f2eade, State: Initialized, Role: LEARNER
I20240501 07:59:19.150955 13767 consensus_queue.cc:260] T 1c32cd00bc344ae1b161d3f9b43b53d3 P 74077b61d6f54342bf9736b4a9f2eade [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { replace: true } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: true } }
I20240501 07:59:19.151305 13767 ts_tablet_manager.cc:1398] T 1c32cd00bc344ae1b161d3f9b43b53d3 P 74077b61d6f54342bf9736b4a9f2eade: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
@ 0x7fb0bac50efc kudu::ThreadPoolBuilder::Build() at ??:0
I20240501 07:59:19.152779 6335 tablet_copy_service.cc:342] P 8b8b3ad3aa8f4053b2167dd550f2086f: Request end of tablet copy session 74077b61d6f54342bf9736b4a9f2eade-1c32cd00bc344ae1b161d3f9b43b53d3 received from {username='slave'} at 127.0.99.133:50111
I20240501 07:59:19.152860 6335 tablet_copy_service.cc:433] P 8b8b3ad3aa8f4053b2167dd550f2086f: ending tablet copy session 74077b61d6f54342bf9736b4a9f2eade-1c32cd00bc344ae1b161d3f9b43b53d3 on tablet 1c32cd00bc344ae1b161d3f9b43b53d3 with peer 74077b61d6f54342bf9736b4a9f2eade
@ 0x7fb0bd80377a kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
@ 0x7fb0bd80ca45 kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
@ 0x7fb0bd88d76f kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
@ 0x7fb0bd8a14f6 kudu::tserver::TabletCopyRunnable::Run() at ??:0
@ 0x7fb0bd88c226 _ZZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS_6StatusENS0_24TabletServerErrorPB_CodeEEEENKUlvE_clEv at ??:0
@ 0x7fb0bd89bc0a _ZNSt17_Function_handlerIFvvEZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS1_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS1_6StatusENS2_24TabletServerErrorPB_CodeEEEEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x558b7286575c std::function<>::operator()() at ??:0
@ 0x7fb0bac55e06 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fb0bac566ef _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7fb0bac5803b _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x558b7286575c std::function<>::operator()() at ??:0
@ 0x7fb0bac45807 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fb0bb4836db start_thread at ??:0
@ 0x7fb0b89bb71f clone at ??:0
I20240501 07:59:19.194336 6168 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55950:
name: "rebalancer_extra_table_30"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 07:59:19.198074 13775 ts_tablet_manager.cc:902] T 0864c290fafa488abefa93d41c0a2efe P 8315d22c56f7439a9e1fc82f153f018c: Initiating tablet copy from peer 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134:40141)
I20240501 07:59:19.198529 13775 tablet_copy_client.cc:308] T 0864c290fafa488abefa93d41c0a2efe P 8315d22c56f7439a9e1fc82f153f018c: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.134:40141
I20240501 07:59:19.200057 8945 tablet_copy_service.cc:140] P 13d05ebdca8c4f489c597ce550dd773d: Received BeginTabletCopySession request for tablet 0864c290fafa488abefa93d41c0a2efe from peer 8315d22c56f7439a9e1fc82f153f018c ({username='slave'} at 127.0.99.130:51073)
I20240501 07:59:19.200177 8945 tablet_copy_service.cc:161] P 13d05ebdca8c4f489c597ce550dd773d: Beginning new tablet copy session on tablet 0864c290fafa488abefa93d41c0a2efe from peer 8315d22c56f7439a9e1fc82f153f018c at {username='slave'} at 127.0.99.130:51073: session id = 8315d22c56f7439a9e1fc82f153f018c-0864c290fafa488abefa93d41c0a2efe
I20240501 07:59:19.201213 8945 tablet_copy_source_session.cc:205] T 0864c290fafa488abefa93d41c0a2efe P 13d05ebdca8c4f489c597ce550dd773d: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:19.202510 13775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0864c290fafa488abefa93d41c0a2efe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:19.203593 8774 tablet_service.cc:1467] Processing CreateTablet for tablet f3375300a7914fa599fe0d3a8f1f9c18 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=e7fc4d6ee10a4578ab351d70176438eb]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:19.203876 8774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3375300a7914fa599fe0d3a8f1f9c18. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:19.204257 9036 tablet_service.cc:1467] Processing CreateTablet for tablet f3375300a7914fa599fe0d3a8f1f9c18 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=e7fc4d6ee10a4578ab351d70176438eb]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:19.204268 9034 tablet_service.cc:1467] Processing CreateTablet for tablet 7dbbfc5d5848486f983cffdd2f36cc91 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=e7fc4d6ee10a4578ab351d70176438eb]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:19.204598 9034 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7dbbfc5d5848486f983cffdd2f36cc91. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:19.204713 9036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3375300a7914fa599fe0d3a8f1f9c18. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:19.205269 8773 tablet_service.cc:1467] Processing CreateTablet for tablet ad3e528eaff349a4b3b03037652b1505 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=e7fc4d6ee10a4578ab351d70176438eb]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:19.205497 8773 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad3e528eaff349a4b3b03037652b1505. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:19.206377 8757 tablet_service.cc:1467] Processing CreateTablet for tablet 7dbbfc5d5848486f983cffdd2f36cc91 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=e7fc4d6ee10a4578ab351d70176438eb]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:19.206607 8757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7dbbfc5d5848486f983cffdd2f36cc91. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:19.206859 13781 tablet_bootstrap.cc:492] T 7dbbfc5d5848486f983cffdd2f36cc91 P a505e7cb133c4392bfda8a4a20874ae9: Bootstrap starting.
I20240501 07:59:19.207737 6687 tablet_service.cc:1467] Processing CreateTablet for tablet f3375300a7914fa599fe0d3a8f1f9c18 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=e7fc4d6ee10a4578ab351d70176438eb]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:19.207846 13781 tablet_bootstrap.cc:654] T 7dbbfc5d5848486f983cffdd2f36cc91 P a505e7cb133c4392bfda8a4a20874ae9: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:19.207996 6687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3375300a7914fa599fe0d3a8f1f9c18. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:19.210346 13781 tablet_bootstrap.cc:492] T 7dbbfc5d5848486f983cffdd2f36cc91 P a505e7cb133c4392bfda8a4a20874ae9: No bootstrap required, opened a new log
I20240501 07:59:19.210426 13781 ts_tablet_manager.cc:1367] T 7dbbfc5d5848486f983cffdd2f36cc91 P a505e7cb133c4392bfda8a4a20874ae9: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 07:59:19.210552 9035 tablet_service.cc:1467] Processing CreateTablet for tablet ad3e528eaff349a4b3b03037652b1505 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=e7fc4d6ee10a4578ab351d70176438eb]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:19.210776 9035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad3e528eaff349a4b3b03037652b1505. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:19.210901 13781 raft_consensus.cc:348] T 7dbbfc5d5848486f983cffdd2f36cc91 P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } }
I20240501 07:59:19.210994 13781 raft_consensus.cc:374] T 7dbbfc5d5848486f983cffdd2f36cc91 P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:19.211035 13781 raft_consensus.cc:729] T 7dbbfc5d5848486f983cffdd2f36cc91 P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a505e7cb133c4392bfda8a4a20874ae9, State: Initialized, Role: FOLLOWER
I20240501 07:59:19.211149 13781 consensus_queue.cc:260] T 7dbbfc5d5848486f983cffdd2f36cc91 P a505e7cb133c4392bfda8a4a20874ae9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } }
I20240501 07:59:19.211400 13781 ts_tablet_manager.cc:1398] T 7dbbfc5d5848486f983cffdd2f36cc91 P a505e7cb133c4392bfda8a4a20874ae9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:19.213323 8905 tablet_service.cc:1467] Processing CreateTablet for tablet ad3e528eaff349a4b3b03037652b1505 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=e7fc4d6ee10a4578ab351d70176438eb]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:19.213570 8905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad3e528eaff349a4b3b03037652b1505. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:19.213948 13856 tablet_bootstrap.cc:492] T f3375300a7914fa599fe0d3a8f1f9c18 P 74077b61d6f54342bf9736b4a9f2eade: Bootstrap starting.
I20240501 07:59:19.214944 13775 tablet_copy_client.cc:783] T 0864c290fafa488abefa93d41c0a2efe P 8315d22c56f7439a9e1fc82f153f018c: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:19.215123 13775 tablet_copy_client.cc:647] T 0864c290fafa488abefa93d41c0a2efe P 8315d22c56f7439a9e1fc82f153f018c: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:19.215293 13781 tablet_bootstrap.cc:492] T ad3e528eaff349a4b3b03037652b1505 P a505e7cb133c4392bfda8a4a20874ae9: Bootstrap starting.
I20240501 07:59:19.215692 13774 tablet_bootstrap.cc:492] T ad3e528eaff349a4b3b03037652b1505 P 13d05ebdca8c4f489c597ce550dd773d: Bootstrap starting.
I20240501 07:59:19.216346 13781 tablet_bootstrap.cc:654] T ad3e528eaff349a4b3b03037652b1505 P a505e7cb133c4392bfda8a4a20874ae9: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:19.216677 13774 tablet_bootstrap.cc:654] T ad3e528eaff349a4b3b03037652b1505 P 13d05ebdca8c4f489c597ce550dd773d: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:19.218412 13775 tablet_copy_client.cc:523] T 0864c290fafa488abefa93d41c0a2efe P 8315d22c56f7439a9e1fc82f153f018c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:19.218879 13856 tablet_bootstrap.cc:654] T f3375300a7914fa599fe0d3a8f1f9c18 P 74077b61d6f54342bf9736b4a9f2eade: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:19.219609 13859 tablet_bootstrap.cc:492] T f3375300a7914fa599fe0d3a8f1f9c18 P 1f9cb72be98941b5a8e504cbe34132b9: Bootstrap starting.
I20240501 07:59:19.219995 13775 tablet_bootstrap.cc:492] T 0864c290fafa488abefa93d41c0a2efe P 8315d22c56f7439a9e1fc82f153f018c: Bootstrap starting.
I20240501 07:59:19.220449 13859 tablet_bootstrap.cc:654] T f3375300a7914fa599fe0d3a8f1f9c18 P 1f9cb72be98941b5a8e504cbe34132b9: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:19.221995 13750 ts_tablet_manager.cc:902] T c16b7962c18d4e02b850c7c7c678eae4 P 8b8b3ad3aa8f4053b2167dd550f2086f: Initiating tablet copy from peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607)
I20240501 07:59:19.222388 13750 tablet_copy_client.cc:308] T c16b7962c18d4e02b850c7c7c678eae4 P 8b8b3ad3aa8f4053b2167dd550f2086f: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.135:42607
I20240501 07:59:19.228118 13781 tablet_bootstrap.cc:492] T ad3e528eaff349a4b3b03037652b1505 P a505e7cb133c4392bfda8a4a20874ae9: No bootstrap required, opened a new log
I20240501 07:59:19.228242 13781 ts_tablet_manager.cc:1367] T ad3e528eaff349a4b3b03037652b1505 P a505e7cb133c4392bfda8a4a20874ae9: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20240501 07:59:19.228670 13781 raft_consensus.cc:348] T ad3e528eaff349a4b3b03037652b1505 P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } }
I20240501 07:59:19.228765 13781 raft_consensus.cc:374] T ad3e528eaff349a4b3b03037652b1505 P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:19.228788 13781 raft_consensus.cc:729] T ad3e528eaff349a4b3b03037652b1505 P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a505e7cb133c4392bfda8a4a20874ae9, State: Initialized, Role: FOLLOWER
I20240501 07:59:19.228890 13781 consensus_queue.cc:260] T ad3e528eaff349a4b3b03037652b1505 P a505e7cb133c4392bfda8a4a20874ae9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } }
I20240501 07:59:19.229184 13781 ts_tablet_manager.cc:1398] T ad3e528eaff349a4b3b03037652b1505 P a505e7cb133c4392bfda8a4a20874ae9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:19.229427 13781 tablet_bootstrap.cc:492] T f3375300a7914fa599fe0d3a8f1f9c18 P a505e7cb133c4392bfda8a4a20874ae9: Bootstrap starting.
I20240501 07:59:19.229939 13774 tablet_bootstrap.cc:492] T ad3e528eaff349a4b3b03037652b1505 P 13d05ebdca8c4f489c597ce550dd773d: No bootstrap required, opened a new log
I20240501 07:59:19.230032 13774 ts_tablet_manager.cc:1367] T ad3e528eaff349a4b3b03037652b1505 P 13d05ebdca8c4f489c597ce550dd773d: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.002s
I20240501 07:59:19.230484 13781 tablet_bootstrap.cc:654] T f3375300a7914fa599fe0d3a8f1f9c18 P a505e7cb133c4392bfda8a4a20874ae9: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:19.230473 13774 raft_consensus.cc:348] T ad3e528eaff349a4b3b03037652b1505 P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } }
I20240501 07:59:19.230602 13774 raft_consensus.cc:374] T ad3e528eaff349a4b3b03037652b1505 P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:19.230648 13774 raft_consensus.cc:729] T ad3e528eaff349a4b3b03037652b1505 P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13d05ebdca8c4f489c597ce550dd773d, State: Initialized, Role: FOLLOWER
I20240501 07:59:19.230748 13774 consensus_queue.cc:260] T ad3e528eaff349a4b3b03037652b1505 P 13d05ebdca8c4f489c597ce550dd773d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } }
I20240501 07:59:19.231036 13774 ts_tablet_manager.cc:1398] T ad3e528eaff349a4b3b03037652b1505 P 13d05ebdca8c4f489c597ce550dd773d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:59:19.231738 13859 tablet_bootstrap.cc:492] T f3375300a7914fa599fe0d3a8f1f9c18 P 1f9cb72be98941b5a8e504cbe34132b9: No bootstrap required, opened a new log
I20240501 07:59:19.232507 9076 tablet_copy_service.cc:140] P a505e7cb133c4392bfda8a4a20874ae9: Received BeginTabletCopySession request for tablet c16b7962c18d4e02b850c7c7c678eae4 from peer 8b8b3ad3aa8f4053b2167dd550f2086f ({username='slave'} at 127.0.99.129:56609)
I20240501 07:59:19.232596 9076 tablet_copy_service.cc:161] P a505e7cb133c4392bfda8a4a20874ae9: Beginning new tablet copy session on tablet c16b7962c18d4e02b850c7c7c678eae4 from peer 8b8b3ad3aa8f4053b2167dd550f2086f at {username='slave'} at 127.0.99.129:56609: session id = 8b8b3ad3aa8f4053b2167dd550f2086f-c16b7962c18d4e02b850c7c7c678eae4
I20240501 07:59:19.233604 9076 tablet_copy_source_session.cc:205] T c16b7962c18d4e02b850c7c7c678eae4 P a505e7cb133c4392bfda8a4a20874ae9: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:19.231842 13859 ts_tablet_manager.cc:1367] T f3375300a7914fa599fe0d3a8f1f9c18 P 1f9cb72be98941b5a8e504cbe34132b9: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240501 07:59:19.235132 13750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c16b7962c18d4e02b850c7c7c678eae4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:19.235080 13859 raft_consensus.cc:348] T f3375300a7914fa599fe0d3a8f1f9c18 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } }
I20240501 07:59:19.235206 13859 raft_consensus.cc:374] T f3375300a7914fa599fe0d3a8f1f9c18 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:19.235244 13859 raft_consensus.cc:729] T f3375300a7914fa599fe0d3a8f1f9c18 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f9cb72be98941b5a8e504cbe34132b9, State: Initialized, Role: FOLLOWER
I20240501 07:59:19.235379 13859 consensus_queue.cc:260] T f3375300a7914fa599fe0d3a8f1f9c18 P 1f9cb72be98941b5a8e504cbe34132b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } }
I20240501 07:59:19.236322 13859 ts_tablet_manager.cc:1398] T f3375300a7914fa599fe0d3a8f1f9c18 P 1f9cb72be98941b5a8e504cbe34132b9: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:59:19.238471 13756 ts_tablet_manager.cc:902] T e5d564b9558946f585ca7a7a3b457a07 P 1f9cb72be98941b5a8e504cbe34132b9: Initiating tablet copy from peer 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129:46875)
I20240501 07:59:19.238973 13756 tablet_copy_client.cc:308] T e5d564b9558946f585ca7a7a3b457a07 P 1f9cb72be98941b5a8e504cbe34132b9: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:46875
I20240501 07:59:19.239490 6335 tablet_copy_service.cc:140] P 8b8b3ad3aa8f4053b2167dd550f2086f: Received BeginTabletCopySession request for tablet e5d564b9558946f585ca7a7a3b457a07 from peer 1f9cb72be98941b5a8e504cbe34132b9 ({username='slave'} at 127.0.99.132:34199)
I20240501 07:59:19.239595 6335 tablet_copy_service.cc:161] P 8b8b3ad3aa8f4053b2167dd550f2086f: Beginning new tablet copy session on tablet e5d564b9558946f585ca7a7a3b457a07 from peer 1f9cb72be98941b5a8e504cbe34132b9 at {username='slave'} at 127.0.99.132:34199: session id = 1f9cb72be98941b5a8e504cbe34132b9-e5d564b9558946f585ca7a7a3b457a07
I20240501 07:59:19.240255 13750 tablet_copy_client.cc:783] T c16b7962c18d4e02b850c7c7c678eae4 P 8b8b3ad3aa8f4053b2167dd550f2086f: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:19.240396 13750 tablet_copy_client.cc:647] T c16b7962c18d4e02b850c7c7c678eae4 P 8b8b3ad3aa8f4053b2167dd550f2086f: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:19.240511 6335 tablet_copy_source_session.cc:205] T e5d564b9558946f585ca7a7a3b457a07 P 8b8b3ad3aa8f4053b2167dd550f2086f: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:19.241132 13775 tablet_bootstrap.cc:492] T 0864c290fafa488abefa93d41c0a2efe P 8315d22c56f7439a9e1fc82f153f018c: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:19.241395 13775 tablet_bootstrap.cc:492] T 0864c290fafa488abefa93d41c0a2efe P 8315d22c56f7439a9e1fc82f153f018c: Bootstrap complete.
I20240501 07:59:19.241559 13756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5d564b9558946f585ca7a7a3b457a07. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:19.241585 13775 ts_tablet_manager.cc:1367] T 0864c290fafa488abefa93d41c0a2efe P 8315d22c56f7439a9e1fc82f153f018c: Time spent bootstrapping tablet: real 0.022s user 0.005s sys 0.000s
I20240501 07:59:19.242066 13775 raft_consensus.cc:348] T 0864c290fafa488abefa93d41c0a2efe P 8315d22c56f7439a9e1fc82f153f018c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { replace: true } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { promote: true } }
I20240501 07:59:19.242182 13775 raft_consensus.cc:729] T 0864c290fafa488abefa93d41c0a2efe P 8315d22c56f7439a9e1fc82f153f018c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8315d22c56f7439a9e1fc82f153f018c, State: Initialized, Role: LEARNER
I20240501 07:59:19.242274 13775 consensus_queue.cc:260] T 0864c290fafa488abefa93d41c0a2efe P 8315d22c56f7439a9e1fc82f153f018c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { replace: true } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { promote: true } }
I20240501 07:59:19.243346 13775 ts_tablet_manager.cc:1398] T 0864c290fafa488abefa93d41c0a2efe P 8315d22c56f7439a9e1fc82f153f018c: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:59:19.243816 8945 tablet_copy_service.cc:342] P 13d05ebdca8c4f489c597ce550dd773d: Request end of tablet copy session 8315d22c56f7439a9e1fc82f153f018c-0864c290fafa488abefa93d41c0a2efe received from {username='slave'} at 127.0.99.130:51073
I20240501 07:59:19.243891 8945 tablet_copy_service.cc:433] P 13d05ebdca8c4f489c597ce550dd773d: ending tablet copy session 8315d22c56f7439a9e1fc82f153f018c-0864c290fafa488abefa93d41c0a2efe on tablet 0864c290fafa488abefa93d41c0a2efe with peer 8315d22c56f7439a9e1fc82f153f018c
I20240501 07:59:19.243898 13756 tablet_copy_client.cc:783] T e5d564b9558946f585ca7a7a3b457a07 P 1f9cb72be98941b5a8e504cbe34132b9: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:19.244030 13756 tablet_copy_client.cc:647] T e5d564b9558946f585ca7a7a3b457a07 P 1f9cb72be98941b5a8e504cbe34132b9: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:19.245842 13756 tablet_copy_client.cc:523] T e5d564b9558946f585ca7a7a3b457a07 P 1f9cb72be98941b5a8e504cbe34132b9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:19.247383 13756 tablet_bootstrap.cc:492] T e5d564b9558946f585ca7a7a3b457a07 P 1f9cb72be98941b5a8e504cbe34132b9: Bootstrap starting.
I20240501 07:59:19.251242 13756 tablet_bootstrap.cc:492] T e5d564b9558946f585ca7a7a3b457a07 P 1f9cb72be98941b5a8e504cbe34132b9: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:19.251504 13756 tablet_bootstrap.cc:492] T e5d564b9558946f585ca7a7a3b457a07 P 1f9cb72be98941b5a8e504cbe34132b9: Bootstrap complete.
I20240501 07:59:19.251663 13756 ts_tablet_manager.cc:1367] T e5d564b9558946f585ca7a7a3b457a07 P 1f9cb72be98941b5a8e504cbe34132b9: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20240501 07:59:19.252177 13756 raft_consensus.cc:348] T e5d564b9558946f585ca7a7a3b457a07 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { replace: true } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { promote: true } }
I20240501 07:59:19.252302 13756 raft_consensus.cc:729] T e5d564b9558946f585ca7a7a3b457a07 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1f9cb72be98941b5a8e504cbe34132b9, State: Initialized, Role: LEARNER
I20240501 07:59:19.252424 13756 consensus_queue.cc:260] T e5d564b9558946f585ca7a7a3b457a07 P 1f9cb72be98941b5a8e504cbe34132b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { replace: true } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { promote: true } }
I20240501 07:59:19.252806 13781 tablet_bootstrap.cc:492] T f3375300a7914fa599fe0d3a8f1f9c18 P a505e7cb133c4392bfda8a4a20874ae9: No bootstrap required, opened a new log
I20240501 07:59:19.252817 13756 ts_tablet_manager.cc:1398] T e5d564b9558946f585ca7a7a3b457a07 P 1f9cb72be98941b5a8e504cbe34132b9: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240501 07:59:19.252899 13781 ts_tablet_manager.cc:1367] T f3375300a7914fa599fe0d3a8f1f9c18 P a505e7cb133c4392bfda8a4a20874ae9: Time spent bootstrapping tablet: real 0.024s user 0.002s sys 0.000s
I20240501 07:59:19.253324 6335 tablet_copy_service.cc:342] P 8b8b3ad3aa8f4053b2167dd550f2086f: Request end of tablet copy session 1f9cb72be98941b5a8e504cbe34132b9-e5d564b9558946f585ca7a7a3b457a07 received from {username='slave'} at 127.0.99.132:34199
I20240501 07:59:19.253412 6335 tablet_copy_service.cc:433] P 8b8b3ad3aa8f4053b2167dd550f2086f: ending tablet copy session 1f9cb72be98941b5a8e504cbe34132b9-e5d564b9558946f585ca7a7a3b457a07 on tablet e5d564b9558946f585ca7a7a3b457a07 with peer 1f9cb72be98941b5a8e504cbe34132b9
I20240501 07:59:19.253418 13781 raft_consensus.cc:348] T f3375300a7914fa599fe0d3a8f1f9c18 P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } }
I20240501 07:59:19.253530 13781 raft_consensus.cc:374] T f3375300a7914fa599fe0d3a8f1f9c18 P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:19.253562 13781 raft_consensus.cc:729] T f3375300a7914fa599fe0d3a8f1f9c18 P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a505e7cb133c4392bfda8a4a20874ae9, State: Initialized, Role: FOLLOWER
I20240501 07:59:19.253683 13781 consensus_queue.cc:260] T f3375300a7914fa599fe0d3a8f1f9c18 P a505e7cb133c4392bfda8a4a20874ae9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } }
I20240501 07:59:19.254040 13781 ts_tablet_manager.cc:1398] T f3375300a7914fa599fe0d3a8f1f9c18 P a505e7cb133c4392bfda8a4a20874ae9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:19.255596 13856 tablet_bootstrap.cc:492] T f3375300a7914fa599fe0d3a8f1f9c18 P 74077b61d6f54342bf9736b4a9f2eade: No bootstrap required, opened a new log
I20240501 07:59:19.255673 13856 ts_tablet_manager.cc:1367] T f3375300a7914fa599fe0d3a8f1f9c18 P 74077b61d6f54342bf9736b4a9f2eade: Time spent bootstrapping tablet: real 0.042s user 0.002s sys 0.000s
I20240501 07:59:19.256039 13856 raft_consensus.cc:348] T f3375300a7914fa599fe0d3a8f1f9c18 P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } }
I20240501 07:59:19.256122 13856 raft_consensus.cc:374] T f3375300a7914fa599fe0d3a8f1f9c18 P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:19.256142 13856 raft_consensus.cc:729] T f3375300a7914fa599fe0d3a8f1f9c18 P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74077b61d6f54342bf9736b4a9f2eade, State: Initialized, Role: FOLLOWER
I20240501 07:59:19.256253 13856 consensus_queue.cc:260] T f3375300a7914fa599fe0d3a8f1f9c18 P 74077b61d6f54342bf9736b4a9f2eade [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } }
I20240501 07:59:19.256537 13856 ts_tablet_manager.cc:1398] T f3375300a7914fa599fe0d3a8f1f9c18 P 74077b61d6f54342bf9736b4a9f2eade: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:19.256740 13856 tablet_bootstrap.cc:492] T ad3e528eaff349a4b3b03037652b1505 P 74077b61d6f54342bf9736b4a9f2eade: Bootstrap starting.
I20240501 07:59:19.257730 13856 tablet_bootstrap.cc:654] T ad3e528eaff349a4b3b03037652b1505 P 74077b61d6f54342bf9736b4a9f2eade: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:19.262504 13750 tablet_copy_client.cc:523] T c16b7962c18d4e02b850c7c7c678eae4 P 8b8b3ad3aa8f4053b2167dd550f2086f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:19.263386 13856 tablet_bootstrap.cc:492] T ad3e528eaff349a4b3b03037652b1505 P 74077b61d6f54342bf9736b4a9f2eade: No bootstrap required, opened a new log
I20240501 07:59:19.263491 13856 ts_tablet_manager.cc:1367] T ad3e528eaff349a4b3b03037652b1505 P 74077b61d6f54342bf9736b4a9f2eade: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240501 07:59:19.263867 13538 raft_consensus.cc:482] T f3375300a7914fa599fe0d3a8f1f9c18 P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:19.263948 13538 raft_consensus.cc:504] T f3375300a7914fa599fe0d3a8f1f9c18 P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } }
I20240501 07:59:19.264040 13856 raft_consensus.cc:348] T ad3e528eaff349a4b3b03037652b1505 P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } }
I20240501 07:59:19.264106 13856 raft_consensus.cc:374] T ad3e528eaff349a4b3b03037652b1505 P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:19.264133 13856 raft_consensus.cc:729] T ad3e528eaff349a4b3b03037652b1505 P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74077b61d6f54342bf9736b4a9f2eade, State: Initialized, Role: FOLLOWER
I20240501 07:59:19.264134 13750 tablet_bootstrap.cc:492] T c16b7962c18d4e02b850c7c7c678eae4 P 8b8b3ad3aa8f4053b2167dd550f2086f: Bootstrap starting.
I20240501 07:59:19.264246 13856 consensus_queue.cc:260] T ad3e528eaff349a4b3b03037652b1505 P 74077b61d6f54342bf9736b4a9f2eade [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } }
I20240501 07:59:19.264401 13538 leader_election.cc:288] T f3375300a7914fa599fe0d3a8f1f9c18 P 74077b61d6f54342bf9736b4a9f2eade [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132:36773), a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607)
I20240501 07:59:19.264575 13856 ts_tablet_manager.cc:1398] T ad3e528eaff349a4b3b03037652b1505 P 74077b61d6f54342bf9736b4a9f2eade: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:19.264732 13856 tablet_bootstrap.cc:492] T 7dbbfc5d5848486f983cffdd2f36cc91 P 74077b61d6f54342bf9736b4a9f2eade: Bootstrap starting.
I20240501 07:59:19.265741 13856 tablet_bootstrap.cc:654] T 7dbbfc5d5848486f983cffdd2f36cc91 P 74077b61d6f54342bf9736b4a9f2eade: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:19.265910 6707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3375300a7914fa599fe0d3a8f1f9c18" candidate_uuid: "74077b61d6f54342bf9736b4a9f2eade" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f9cb72be98941b5a8e504cbe34132b9" is_pre_election: true
I20240501 07:59:19.266041 6707 raft_consensus.cc:2454] T f3375300a7914fa599fe0d3a8f1f9c18 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74077b61d6f54342bf9736b4a9f2eade in term 0.
I20240501 07:59:19.266338 8729 leader_election.cc:302] T f3375300a7914fa599fe0d3a8f1f9c18 P 74077b61d6f54342bf9736b4a9f2eade [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f9cb72be98941b5a8e504cbe34132b9, 74077b61d6f54342bf9736b4a9f2eade; no voters:
I20240501 07:59:19.266607 13218 raft_consensus.cc:2781] T f3375300a7914fa599fe0d3a8f1f9c18 P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:19.266665 13218 raft_consensus.cc:482] T f3375300a7914fa599fe0d3a8f1f9c18 P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:19.266726 13218 raft_consensus.cc:3037] T f3375300a7914fa599fe0d3a8f1f9c18 P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:19.267833 13218 raft_consensus.cc:504] T f3375300a7914fa599fe0d3a8f1f9c18 P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } }
I20240501 07:59:19.268082 13789 ts_tablet_manager.cc:902] T 8f36eefcda3f475d9571e92fdeb52eb4 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Initiating tablet copy from peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607)
I20240501 07:59:19.268227 13218 leader_election.cc:288] T f3375300a7914fa599fe0d3a8f1f9c18 P 74077b61d6f54342bf9736b4a9f2eade [CANDIDATE]: Term 1 election: Requested vote from peers 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132:36773), a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607)
I20240501 07:59:19.268543 13789 tablet_copy_client.cc:308] T 8f36eefcda3f475d9571e92fdeb52eb4 P 3b164cbbc00f4e8eb3254ddd349a3ff9: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.135:42607
I20240501 07:59:19.269016 9056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3375300a7914fa599fe0d3a8f1f9c18" candidate_uuid: "74077b61d6f54342bf9736b4a9f2eade" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a505e7cb133c4392bfda8a4a20874ae9" is_pre_election: true
I20240501 07:59:19.269141 9056 raft_consensus.cc:2454] T f3375300a7914fa599fe0d3a8f1f9c18 P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74077b61d6f54342bf9736b4a9f2eade in term 0.
I20240501 07:59:19.269201 6707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3375300a7914fa599fe0d3a8f1f9c18" candidate_uuid: "74077b61d6f54342bf9736b4a9f2eade" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f9cb72be98941b5a8e504cbe34132b9"
I20240501 07:59:19.269310 6707 raft_consensus.cc:3037] T f3375300a7914fa599fe0d3a8f1f9c18 P 1f9cb72be98941b5a8e504cbe34132b9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:19.269325 9055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3375300a7914fa599fe0d3a8f1f9c18" candidate_uuid: "74077b61d6f54342bf9736b4a9f2eade" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a505e7cb133c4392bfda8a4a20874ae9"
I20240501 07:59:19.269407 9055 raft_consensus.cc:3037] T f3375300a7914fa599fe0d3a8f1f9c18 P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:19.270323 6707 raft_consensus.cc:2454] T f3375300a7914fa599fe0d3a8f1f9c18 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74077b61d6f54342bf9736b4a9f2eade in term 1.
I20240501 07:59:19.270323 9055 raft_consensus.cc:2454] T f3375300a7914fa599fe0d3a8f1f9c18 P a505e7cb133c4392bfda8a4a20874ae9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74077b61d6f54342bf9736b4a9f2eade in term 1.
I20240501 07:59:19.270721 8728 leader_election.cc:302] T f3375300a7914fa599fe0d3a8f1f9c18 P 74077b61d6f54342bf9736b4a9f2eade [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74077b61d6f54342bf9736b4a9f2eade, a505e7cb133c4392bfda8a4a20874ae9; no voters:
I20240501 07:59:19.271169 9076 tablet_copy_service.cc:140] P a505e7cb133c4392bfda8a4a20874ae9: Received BeginTabletCopySession request for tablet 8f36eefcda3f475d9571e92fdeb52eb4 from peer 3b164cbbc00f4e8eb3254ddd349a3ff9 ({username='slave'} at 127.0.99.131:39545)
I20240501 07:59:19.271250 9076 tablet_copy_service.cc:161] P a505e7cb133c4392bfda8a4a20874ae9: Beginning new tablet copy session on tablet 8f36eefcda3f475d9571e92fdeb52eb4 from peer 3b164cbbc00f4e8eb3254ddd349a3ff9 at {username='slave'} at 127.0.99.131:39545: session id = 3b164cbbc00f4e8eb3254ddd349a3ff9-8f36eefcda3f475d9571e92fdeb52eb4
I20240501 07:59:19.271829 13856 tablet_bootstrap.cc:492] T 7dbbfc5d5848486f983cffdd2f36cc91 P 74077b61d6f54342bf9736b4a9f2eade: No bootstrap required, opened a new log
I20240501 07:59:19.271908 13856 ts_tablet_manager.cc:1367] T 7dbbfc5d5848486f983cffdd2f36cc91 P 74077b61d6f54342bf9736b4a9f2eade: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240501 07:59:19.272264 9076 tablet_copy_source_session.cc:205] T 8f36eefcda3f475d9571e92fdeb52eb4 P a505e7cb133c4392bfda8a4a20874ae9: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:19.272373 13856 raft_consensus.cc:348] T 7dbbfc5d5848486f983cffdd2f36cc91 P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } }
I20240501 07:59:19.272480 13856 raft_consensus.cc:374] T 7dbbfc5d5848486f983cffdd2f36cc91 P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:19.272512 13856 raft_consensus.cc:729] T 7dbbfc5d5848486f983cffdd2f36cc91 P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74077b61d6f54342bf9736b4a9f2eade, State: Initialized, Role: FOLLOWER
I20240501 07:59:19.272632 13856 consensus_queue.cc:260] T 7dbbfc5d5848486f983cffdd2f36cc91 P 74077b61d6f54342bf9736b4a9f2eade [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } }
I20240501 07:59:19.272927 13856 ts_tablet_manager.cc:1398] T 7dbbfc5d5848486f983cffdd2f36cc91 P 74077b61d6f54342bf9736b4a9f2eade: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:19.273202 13538 raft_consensus.cc:2781] T f3375300a7914fa599fe0d3a8f1f9c18 P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:19.273284 13538 raft_consensus.cc:686] T f3375300a7914fa599fe0d3a8f1f9c18 P 74077b61d6f54342bf9736b4a9f2eade [term 1 LEADER]: Becoming Leader. State: Replica: 74077b61d6f54342bf9736b4a9f2eade, State: Running, Role: LEADER
I20240501 07:59:19.273327 13789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f36eefcda3f475d9571e92fdeb52eb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:19.273402 13538 consensus_queue.cc:237] T f3375300a7914fa599fe0d3a8f1f9c18 P 74077b61d6f54342bf9736b4a9f2eade [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } }
I20240501 07:59:19.275712 13789 tablet_copy_client.cc:783] T 8f36eefcda3f475d9571e92fdeb52eb4 P 3b164cbbc00f4e8eb3254ddd349a3ff9: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:19.275869 13789 tablet_copy_client.cc:647] T 8f36eefcda3f475d9571e92fdeb52eb4 P 3b164cbbc00f4e8eb3254ddd349a3ff9: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:19.277597 6164 catalog_manager.cc:5484] T f3375300a7914fa599fe0d3a8f1f9c18 P 74077b61d6f54342bf9736b4a9f2eade reported cstate change: term changed from 0 to 1, leader changed from <none> to 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133). New cstate: current_term: 1 leader_uuid: "74077b61d6f54342bf9736b4a9f2eade" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } health_report { overall_health: UNKNOWN } } }
I20240501 07:59:19.280534 13750 tablet_bootstrap.cc:492] T c16b7962c18d4e02b850c7c7c678eae4 P 8b8b3ad3aa8f4053b2167dd550f2086f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:19.280794 13750 tablet_bootstrap.cc:492] T c16b7962c18d4e02b850c7c7c678eae4 P 8b8b3ad3aa8f4053b2167dd550f2086f: Bootstrap complete.
I20240501 07:59:19.280929 13750 ts_tablet_manager.cc:1367] T c16b7962c18d4e02b850c7c7c678eae4 P 8b8b3ad3aa8f4053b2167dd550f2086f: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.000s
I20240501 07:59:19.281265 13789 tablet_copy_client.cc:523] T 8f36eefcda3f475d9571e92fdeb52eb4 P 3b164cbbc00f4e8eb3254ddd349a3ff9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:19.282749 13789 tablet_bootstrap.cc:492] T 8f36eefcda3f475d9571e92fdeb52eb4 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Bootstrap starting.
I20240501 07:59:19.285729 13750 raft_consensus.cc:348] T c16b7962c18d4e02b850c7c7c678eae4 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { promote: true } }
I20240501 07:59:19.285877 13750 raft_consensus.cc:729] T c16b7962c18d4e02b850c7c7c678eae4 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8b8b3ad3aa8f4053b2167dd550f2086f, State: Initialized, Role: LEARNER
I20240501 07:59:19.286041 13750 consensus_queue.cc:260] T c16b7962c18d4e02b850c7c7c678eae4 P 8b8b3ad3aa8f4053b2167dd550f2086f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { promote: true } }
I20240501 07:59:19.287878 13750 ts_tablet_manager.cc:1398] T c16b7962c18d4e02b850c7c7c678eae4 P 8b8b3ad3aa8f4053b2167dd550f2086f: Time spent starting tablet: real 0.007s user 0.000s sys 0.001s
I20240501 07:59:19.288489 9076 tablet_copy_service.cc:342] P a505e7cb133c4392bfda8a4a20874ae9: Request end of tablet copy session 8b8b3ad3aa8f4053b2167dd550f2086f-c16b7962c18d4e02b850c7c7c678eae4 received from {username='slave'} at 127.0.99.129:56609
I20240501 07:59:19.288585 9076 tablet_copy_service.cc:433] P a505e7cb133c4392bfda8a4a20874ae9: ending tablet copy session 8b8b3ad3aa8f4053b2167dd550f2086f-c16b7962c18d4e02b850c7c7c678eae4 on tablet c16b7962c18d4e02b850c7c7c678eae4 with peer 8b8b3ad3aa8f4053b2167dd550f2086f
I20240501 07:59:19.295295 13901 ts_tablet_manager.cc:902] T a29720541b7b4a79aaa810592ef7c4b1 P a505e7cb133c4392bfda8a4a20874ae9: Initiating tablet copy from peer 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132:36773)
F20240501 07:59:19.295480 13901 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-114b6dbafa1c4c44ac42f1d2293e3125.queue_time_us, tablet-download-pool-114b6dbafa1c4c44ac42f1d2293e3125.queue_time_us) (tablet-download-pool-114b6dbafa1c4c44ac42f1d2293e3125.run_cpu_time_us, tablet-download-pool-114b6dbafa1c4c44ac42f1d2293e3125.run_cpu_time_us) (tablet-download-pool-114b6dbafa1c4c44ac42f1d2293e3125.run_wall_time_us, tablet-download-pool-114b6dbafa1c4c44ac42f1d2293e3125.run_wall_time_us) (tablet-download-pool-26713549654249b09f820a31ee19e2b7.queue_time_us, tablet-download-pool-26713549654249b09f820a31ee19e2b7.queue_time_us) (tablet-download-pool-26713549654249b09f820a31ee19e2b7.run_cpu_time_us, tablet-download-pool-26713549654249b09f820a31ee19e2b7.run_cpu_time_us) (tablet-download-pool-26713549654249b09f820a31ee19e2b7.run_wall_time_us, tablet-download-pool-26713549654249b09f820a31ee19e2b7.run_wall_time_us) (tablet-download-pool-33b3834db0cd4407ae7c26e044459e87.queue_time_us, tablet-download-pool-33b3834db0cd4407ae7c26e044459e87.queue_time_us) (tablet-download-pool-33b3834db0cd4407ae7c26e044459e87.run_cpu_time_us, tablet-download-pool-33b3834db0cd4407ae7c26e044459e87.run_cpu_time_us) (tablet-download-pool-33b3834db0cd4407ae7c26e044459e87.run_wall_time_us, tablet-download-pool-33b3834db0cd4407ae7c26e044459e87.run_wall_time_us) (tablet-download-pool-3745acee2b5543b697a3c8f45d2618bb.queue_time_us, tablet-download-pool-3745acee2b5543b697a3c8f45d2618bb.queue_time_us) (tablet-download-pool-3745acee2b5543b697a3c8f45d2618bb.run_cpu_time_us, tablet-download-pool-3745acee2b5543b697a3c8f45d2618bb.run_cpu_time_us) (tablet-download-pool-3745acee2b5543b697a3c8f45d2618bb.run_wall_time_us, tablet-download-pool-3745acee2b5543b697a3c8f45d2618bb.run_wall_time_us) (tablet-download-pool-3b9b71d44f754353b2321e8c36dd4dfd.queue_time_us, tablet-download-pool-3b9b71d44f754353b2321e8c36dd4dfd.queue_time_us) (tablet-download-pool-3b9b71d44f754353b2321e8c36dd4dfd.run_cpu_time_us, tablet-download-pool-3b9b71d44f754353b2321e8c36dd4dfd.run_cpu_time_us) (tablet-download-pool-3b9b71d44f754353b2321e8c36dd4dfd.run_wall_time_us, tablet-download-pool-3b9b71d44f754353b2321e8c36dd4dfd.run_wall_time_us) (tablet-download-pool-6c3c0d6fe41f4ae196ae29beb9b68125.queue_time_us, tablet-download-pool-6c3c0d6fe41f4ae196ae29beb9b68125.queue_time_us) (tablet-download-pool-6c3c0d6fe41f4ae196ae29beb9b68125.run_cpu_time_us, tablet-download-pool-6c3c0d6fe41f4ae196ae29beb9b68125.run_cpu_time_us) (tablet-download-pool-6c3c0d6fe41f4ae196ae29beb9b68125.run_wall_time_us, tablet-download-pool-6c3c0d6fe41f4ae196ae29beb9b68125.run_wall_time_us) (tablet-download-pool-782e37a4ebb34a68b461a74115ae8703.queue_time_us, tablet-download-pool-782e37a4ebb34a68b461a74115ae8703.queue_time_us) (tablet-download-pool-782e37a4ebb34a68b461a74115ae8703.run_cpu_time_us, tablet-download-pool-782e37a4ebb34a68b461a74115ae8703.run_cpu_time_us) (tablet-download-pool-782e37a4ebb34a68b461a74115ae8703.run_wall_time_us, tablet-download-pool-782e37a4ebb34a68b461a74115ae8703.run_wall_time_us) (tablet-download-pool-810a2791eb04415dae47e85db87352cb.queue_time_us, tablet-download-pool-810a2791eb04415dae47e85db87352cb.queue_time_us) (tablet-download-pool-810a2791eb04415dae47e85db87352cb.run_cpu_time_us, tablet-download-pool-810a2791eb04415dae47e85db87352cb.run_cpu_time_us) (tablet-download-pool-810a2791eb04415dae47e85db87352cb.run_wall_time_us, tablet-download-pool-810a2791eb04415dae47e85db87352cb.run_wall_time_us) (tablet-download-pool-8454f24b0c28451b8ee51be39ca3e07f.queue_time_us, tablet-download-pool-8454f24b0c28451b8ee51be39ca3e07f.queue_time_us) (tablet-download-pool-8454f24b0c28451b8ee51be39ca3e07f.run_cpu_time_us, tablet-download-pool-8454f24b0c28451b8ee51be39ca3e07f.run_cpu_time_us) (tablet-download-pool-8454f24b0c28451b8ee51be39ca3e07f.run_wall_time_us, tablet-download-pool-8454f24b0c28451b8ee51be39ca3e07f.run_wall_time_us) (tablet-download-pool-a29720541b7b4a79aaa810592ef7c4b1.queue_time_us, tablet-download-pool-a29720541b7b4a79aaa810592ef7c4b1.queue_time_us) (tablet-download-pool-a756205204d146ce82a2799e30582f57.queue_time_us, tablet-download-pool-a756205204d146ce82a2799e30582f57.queue_time_us) (tablet-download-pool-a756205204d146ce82a2799e30582f57.run_cpu_time_us, tablet-download-pool-a756205204d146ce82a2799e30582f57.run_cpu_time_us) (tablet-download-pool-a756205204d146ce82a2799e30582f57.run_wall_time_us, tablet-download-pool-a756205204d146ce82a2799e30582f57.run_wall_time_us) (tablet-download-pool-adf207c019994dd49671bab4145fc268.queue_time_us, tablet-download-pool-adf207c019994dd49671bab4145fc268.queue_time_us) (tablet-download-pool-adf207c019994dd49671bab4145fc268.run_cpu_time_us, tablet-download-pool-adf207c019994dd49671bab4145fc268.run_cpu_time_us) (tablet-download-pool-adf207c019994dd49671bab4145fc268.run_wall_time_us, tablet-download-pool-adf207c019994dd49671bab4145fc268.run_wall_time_us) (tablet-download-pool-ae46990350284295bf14aa64d4b6c93e.queue_time_us, tablet-download-pool-ae46990350284295bf14aa64d4b6c93e.queue_time_us) (tablet-download-pool-ae46990350284295bf14aa64d4b6c93e.run_cpu_time_us, tablet-download-pool-ae46990350284295bf14aa64d4b6c93e.run_cpu_time_us) (tablet-download-pool-ae46990350284295bf14aa64d4b6c93e.run_wall_time_us, tablet-download-pool-ae46990350284295bf14aa64d4b6c93e.run_wall_time_us) (tablet-download-pool-d3f2250a62c6429e886753aaffcbe883.queue_time_us, tablet-download-pool-d3f2250a62c6429e886753aaffcbe883.queue_time_us) (tablet-download-pool-d3f2250a62c6429e886753aaffcbe883.run_cpu_time_us, tablet-download-pool-d3f2250a62c6429e886753aaffcbe883.run_cpu_time_us) (tablet-download-pool-d3f2250a62c6429e886753aaffcbe883.run_wall_time_us, tablet-download-pool-d3f2250a62c6429e886753aaffcbe883.run_wall_time_us) (tablet-download-pool-d6c28cd91f624673b8e428f471a1dcce.queue_time_us, tablet-download-pool-d6c28cd91f624673b8e428f471a1dcce.queue_time_us) (tablet-download-pool-d6c28cd91f624673b8e428f471a1dcce.run_cpu_time_us, tablet-download-pool-d6c28cd91f624673b8e428f471a1dcce.run_cpu_time_us) (tablet-download-pool-d6c28cd91f624673b8e428f471a1dcce.run_wall_time_us, tablet-download-pool-d6c28cd91f624673b8e428f471a1dcce.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
I20240501 07:59:19.297497 13789 tablet_bootstrap.cc:492] T 8f36eefcda3f475d9571e92fdeb52eb4 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
*** Check failure stack trace: ***
I20240501 07:59:19.329141 13789 tablet_bootstrap.cc:492] T 8f36eefcda3f475d9571e92fdeb52eb4 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Bootstrap complete.
I20240501 07:59:19.329385 13789 ts_tablet_manager.cc:1367] T 8f36eefcda3f475d9571e92fdeb52eb4 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Time spent bootstrapping tablet: real 0.047s user 0.002s sys 0.003s
I20240501 07:59:19.330011 13789 raft_consensus.cc:348] T 8f36eefcda3f475d9571e92fdeb52eb4 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { replace: true } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: true } }
I20240501 07:59:19.330168 13789 raft_consensus.cc:729] T 8f36eefcda3f475d9571e92fdeb52eb4 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3b164cbbc00f4e8eb3254ddd349a3ff9, State: Initialized, Role: LEARNER
I20240501 07:59:19.330293 13789 consensus_queue.cc:260] T 8f36eefcda3f475d9571e92fdeb52eb4 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { replace: true } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: true } }
I20240501 07:59:19.330662 13789 ts_tablet_manager.cc:1398] T 8f36eefcda3f475d9571e92fdeb52eb4 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:19.331629 9076 tablet_copy_service.cc:342] P a505e7cb133c4392bfda8a4a20874ae9: Request end of tablet copy session 3b164cbbc00f4e8eb3254ddd349a3ff9-8f36eefcda3f475d9571e92fdeb52eb4 received from {username='slave'} at 127.0.99.131:39545
I20240501 07:59:19.331725 9076 tablet_copy_service.cc:433] P a505e7cb133c4392bfda8a4a20874ae9: ending tablet copy session 3b164cbbc00f4e8eb3254ddd349a3ff9-8f36eefcda3f475d9571e92fdeb52eb4 on tablet 8f36eefcda3f475d9571e92fdeb52eb4 with peer 3b164cbbc00f4e8eb3254ddd349a3ff9
@ 0x7f1518e30dcd google::LogMessage::Fail() at ??:0
@ 0x7f1518e34b93 google::LogMessage::SendToLog() at ??:0
@ 0x7f1518e307cc google::LogMessage::Flush() at ??:0
@ 0x7f1518e31f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f151a29a044 kudu::TraceMetrics::InternName() at ??:0
@ 0x7f151a27ef98 kudu::ThreadPool::ThreadPool() at ??:0
I20240501 07:59:19.353214 13756 ts_tablet_manager.cc:902] T 12ce6a21d06c4ba889542758be9dc62d P 1f9cb72be98941b5a8e504cbe34132b9: Initiating tablet copy from peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607)
I20240501 07:59:19.353711 13756 tablet_copy_client.cc:308] T 12ce6a21d06c4ba889542758be9dc62d P 1f9cb72be98941b5a8e504cbe34132b9: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.135:42607
I20240501 07:59:19.355584 9076 tablet_copy_service.cc:140] P a505e7cb133c4392bfda8a4a20874ae9: Received BeginTabletCopySession request for tablet 12ce6a21d06c4ba889542758be9dc62d from peer 1f9cb72be98941b5a8e504cbe34132b9 ({username='slave'} at 127.0.99.132:45983)
I20240501 07:59:19.355696 9076 tablet_copy_service.cc:161] P a505e7cb133c4392bfda8a4a20874ae9: Beginning new tablet copy session on tablet 12ce6a21d06c4ba889542758be9dc62d from peer 1f9cb72be98941b5a8e504cbe34132b9 at {username='slave'} at 127.0.99.132:45983: session id = 1f9cb72be98941b5a8e504cbe34132b9-12ce6a21d06c4ba889542758be9dc62d
I20240501 07:59:19.356134 13750 ts_tablet_manager.cc:902] T 0f9e2549fe294a93b3b25081a15b2028 P 8b8b3ad3aa8f4053b2167dd550f2086f: Initiating tablet copy from peer 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133:39859)
I20240501 07:59:19.356554 13750 tablet_copy_client.cc:308] T 0f9e2549fe294a93b3b25081a15b2028 P 8b8b3ad3aa8f4053b2167dd550f2086f: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.133:39859
I20240501 07:59:19.356832 9076 tablet_copy_source_session.cc:205] T 12ce6a21d06c4ba889542758be9dc62d P a505e7cb133c4392bfda8a4a20874ae9: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:19.357117 8814 tablet_copy_service.cc:140] P 74077b61d6f54342bf9736b4a9f2eade: Received BeginTabletCopySession request for tablet 0f9e2549fe294a93b3b25081a15b2028 from peer 8b8b3ad3aa8f4053b2167dd550f2086f ({username='slave'} at 127.0.99.129:56605)
I20240501 07:59:19.357223 8814 tablet_copy_service.cc:161] P 74077b61d6f54342bf9736b4a9f2eade: Beginning new tablet copy session on tablet 0f9e2549fe294a93b3b25081a15b2028 from peer 8b8b3ad3aa8f4053b2167dd550f2086f at {username='slave'} at 127.0.99.129:56605: session id = 8b8b3ad3aa8f4053b2167dd550f2086f-0f9e2549fe294a93b3b25081a15b2028
I20240501 07:59:19.357905 13756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12ce6a21d06c4ba889542758be9dc62d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:19.358156 8814 tablet_copy_source_session.cc:205] T 0f9e2549fe294a93b3b25081a15b2028 P 74077b61d6f54342bf9736b4a9f2eade: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 07:59:19.359084 13750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f9e2549fe294a93b3b25081a15b2028. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:19.360502 13756 tablet_copy_client.cc:783] T 12ce6a21d06c4ba889542758be9dc62d P 1f9cb72be98941b5a8e504cbe34132b9: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:19.360661 13756 tablet_copy_client.cc:647] T 12ce6a21d06c4ba889542758be9dc62d P 1f9cb72be98941b5a8e504cbe34132b9: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:19.361515 13750 tablet_copy_client.cc:783] T 0f9e2549fe294a93b3b25081a15b2028 P 8b8b3ad3aa8f4053b2167dd550f2086f: tablet copy: Starting download of 0 data blocks...
I20240501 07:59:19.361670 13750 tablet_copy_client.cc:647] T 0f9e2549fe294a93b3b25081a15b2028 P 8b8b3ad3aa8f4053b2167dd550f2086f: tablet copy: Starting download of 1 WAL segments...
I20240501 07:59:19.361876 13756 tablet_copy_client.cc:523] T 12ce6a21d06c4ba889542758be9dc62d P 1f9cb72be98941b5a8e504cbe34132b9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:19.363497 13750 tablet_copy_client.cc:523] T 0f9e2549fe294a93b3b25081a15b2028 P 8b8b3ad3aa8f4053b2167dd550f2086f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 07:59:19.364432 13756 tablet_bootstrap.cc:492] T 12ce6a21d06c4ba889542758be9dc62d P 1f9cb72be98941b5a8e504cbe34132b9: Bootstrap starting.
@ 0x7f151a27cefc kudu::ThreadPoolBuilder::Build() at ??:0
I20240501 07:59:19.364778 13750 tablet_bootstrap.cc:492] T 0f9e2549fe294a93b3b25081a15b2028 P 8b8b3ad3aa8f4053b2167dd550f2086f: Bootstrap starting.
@ 0x7f151ce2f77a kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
I20240501 07:59:19.369938 13750 tablet_bootstrap.cc:492] T 0f9e2549fe294a93b3b25081a15b2028 P 8b8b3ad3aa8f4053b2167dd550f2086f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:19.370194 13750 tablet_bootstrap.cc:492] T 0f9e2549fe294a93b3b25081a15b2028 P 8b8b3ad3aa8f4053b2167dd550f2086f: Bootstrap complete.
I20240501 07:59:19.370357 13750 ts_tablet_manager.cc:1367] T 0f9e2549fe294a93b3b25081a15b2028 P 8b8b3ad3aa8f4053b2167dd550f2086f: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240501 07:59:19.370901 13756 tablet_bootstrap.cc:492] T 12ce6a21d06c4ba889542758be9dc62d P 1f9cb72be98941b5a8e504cbe34132b9: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 07:59:19.370970 13750 raft_consensus.cc:348] T 0f9e2549fe294a93b3b25081a15b2028 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { promote: true } }
I20240501 07:59:19.371110 13750 raft_consensus.cc:729] T 0f9e2549fe294a93b3b25081a15b2028 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8b8b3ad3aa8f4053b2167dd550f2086f, State: Initialized, Role: LEARNER
I20240501 07:59:19.371142 13756 tablet_bootstrap.cc:492] T 12ce6a21d06c4ba889542758be9dc62d P 1f9cb72be98941b5a8e504cbe34132b9: Bootstrap complete.
I20240501 07:59:19.371222 13750 consensus_queue.cc:260] T 0f9e2549fe294a93b3b25081a15b2028 P 8b8b3ad3aa8f4053b2167dd550f2086f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { promote: true } }
I20240501 07:59:19.371290 13756 ts_tablet_manager.cc:1367] T 12ce6a21d06c4ba889542758be9dc62d P 1f9cb72be98941b5a8e504cbe34132b9: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.003s
I20240501 07:59:19.371935 13750 ts_tablet_manager.cc:1398] T 0f9e2549fe294a93b3b25081a15b2028 P 8b8b3ad3aa8f4053b2167dd550f2086f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
@ 0x7f151ce38a45 kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
I20240501 07:59:19.372613 8814 tablet_copy_service.cc:342] P 74077b61d6f54342bf9736b4a9f2eade: Request end of tablet copy session 8b8b3ad3aa8f4053b2167dd550f2086f-0f9e2549fe294a93b3b25081a15b2028 received from {username='slave'} at 127.0.99.129:56605
I20240501 07:59:19.372701 8814 tablet_copy_service.cc:433] P 74077b61d6f54342bf9736b4a9f2eade: ending tablet copy session 8b8b3ad3aa8f4053b2167dd550f2086f-0f9e2549fe294a93b3b25081a15b2028 on tablet 0f9e2549fe294a93b3b25081a15b2028 with peer 8b8b3ad3aa8f4053b2167dd550f2086f
I20240501 07:59:19.372774 13756 raft_consensus.cc:348] T 12ce6a21d06c4ba889542758be9dc62d P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { replace: true } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: false } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { promote: true } }
I20240501 07:59:19.372891 13756 raft_consensus.cc:729] T 12ce6a21d06c4ba889542758be9dc62d P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1f9cb72be98941b5a8e504cbe34132b9, State: Initialized, Role: LEARNER
I20240501 07:59:19.373008 13756 consensus_queue.cc:260] T 12ce6a21d06c4ba889542758be9dc62d P 1f9cb72be98941b5a8e504cbe34132b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { replace: true } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: false } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { promote: true } }
I20240501 07:59:19.373363 13756 ts_tablet_manager.cc:1398] T 12ce6a21d06c4ba889542758be9dc62d P 1f9cb72be98941b5a8e504cbe34132b9: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20240501 07:59:19.373848 9076 tablet_copy_service.cc:342] P a505e7cb133c4392bfda8a4a20874ae9: Request end of tablet copy session 1f9cb72be98941b5a8e504cbe34132b9-12ce6a21d06c4ba889542758be9dc62d received from {username='slave'} at 127.0.99.132:45983
I20240501 07:59:19.373924 9076 tablet_copy_service.cc:433] P a505e7cb133c4392bfda8a4a20874ae9: ending tablet copy session 1f9cb72be98941b5a8e504cbe34132b9-12ce6a21d06c4ba889542758be9dc62d on tablet 12ce6a21d06c4ba889542758be9dc62d with peer 1f9cb72be98941b5a8e504cbe34132b9
@ 0x7f151ceb976f kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
@ 0x7f151cecd4f6 kudu::tserver::TabletCopyRunnable::Run() at ??:0
I20240501 07:59:19.390929 13545 raft_consensus.cc:482] T 7dbbfc5d5848486f983cffdd2f36cc91 P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:19.391058 13545 raft_consensus.cc:504] T 7dbbfc5d5848486f983cffdd2f36cc91 P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } }
@ 0x7f151ceb8226 _ZZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS_6StatusENS0_24TabletServerErrorPB_CodeEEEENKUlvE_clEv at ??:0
I20240501 07:59:19.391543 13545 leader_election.cc:288] T 7dbbfc5d5848486f983cffdd2f36cc91 P a505e7cb133c4392bfda8a4a20874ae9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133:39859), 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149)
I20240501 07:59:19.391906 8794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dbbfc5d5848486f983cffdd2f36cc91" candidate_uuid: "a505e7cb133c4392bfda8a4a20874ae9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74077b61d6f54342bf9736b4a9f2eade" is_pre_election: true
I20240501 07:59:19.392068 8794 raft_consensus.cc:2454] T 7dbbfc5d5848486f983cffdd2f36cc91 P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a505e7cb133c4392bfda8a4a20874ae9 in term 0.
I20240501 07:59:19.392413 8989 leader_election.cc:302] T 7dbbfc5d5848486f983cffdd2f36cc91 P a505e7cb133c4392bfda8a4a20874ae9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74077b61d6f54342bf9736b4a9f2eade, a505e7cb133c4392bfda8a4a20874ae9; no voters:
I20240501 07:59:19.392632 13545 raft_consensus.cc:2781] T 7dbbfc5d5848486f983cffdd2f36cc91 P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:19.392685 13545 raft_consensus.cc:482] T 7dbbfc5d5848486f983cffdd2f36cc91 P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:19.392736 13545 raft_consensus.cc:3037] T 7dbbfc5d5848486f983cffdd2f36cc91 P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:19.394137 13545 raft_consensus.cc:504] T 7dbbfc5d5848486f983cffdd2f36cc91 P a505e7cb133c4392bfda8a4a20874ae9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } }
I20240501 07:59:19.394603 13545 leader_election.cc:288] T 7dbbfc5d5848486f983cffdd2f36cc91 P a505e7cb133c4392bfda8a4a20874ae9 [CANDIDATE]: Term 1 election: Requested vote from peers 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133:39859), 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149)
I20240501 07:59:19.395184 8794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dbbfc5d5848486f983cffdd2f36cc91" candidate_uuid: "a505e7cb133c4392bfda8a4a20874ae9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74077b61d6f54342bf9736b4a9f2eade"
I20240501 07:59:19.395327 8794 raft_consensus.cc:3037] T 7dbbfc5d5848486f983cffdd2f36cc91 P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:19.396560 8794 raft_consensus.cc:2454] T 7dbbfc5d5848486f983cffdd2f36cc91 P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a505e7cb133c4392bfda8a4a20874ae9 in term 1.
I20240501 07:59:19.397017 8989 leader_election.cc:302] T 7dbbfc5d5848486f983cffdd2f36cc91 P a505e7cb133c4392bfda8a4a20874ae9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74077b61d6f54342bf9736b4a9f2eade, a505e7cb133c4392bfda8a4a20874ae9; no voters:
I20240501 07:59:19.397207 13545 raft_consensus.cc:2781] T 7dbbfc5d5848486f983cffdd2f36cc91 P a505e7cb133c4392bfda8a4a20874ae9 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:19.397301 13545 raft_consensus.cc:686] T 7dbbfc5d5848486f983cffdd2f36cc91 P a505e7cb133c4392bfda8a4a20874ae9 [term 1 LEADER]: Becoming Leader. State: Replica: a505e7cb133c4392bfda8a4a20874ae9, State: Running, Role: LEADER
I20240501 07:59:19.397469 13545 consensus_queue.cc:237] T 7dbbfc5d5848486f983cffdd2f36cc91 P a505e7cb133c4392bfda8a4a20874ae9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } }
I20240501 07:59:19.400986 6164 catalog_manager.cc:5484] T 7dbbfc5d5848486f983cffdd2f36cc91 P a505e7cb133c4392bfda8a4a20874ae9 reported cstate change: term changed from 0 to 1, leader changed from <none> to a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135). New cstate: current_term: 1 leader_uuid: "a505e7cb133c4392bfda8a4a20874ae9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } health_report { overall_health: HEALTHY } } }
@ 0x7f151cec7c0a _ZNSt17_Function_handlerIFvvEZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS1_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS1_6StatusENS2_24TabletServerErrorPB_CodeEEEEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x55d26457c75c std::function<>::operator()() at ??:0
@ 0x7f151a281e06 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f151a2826ef _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f151a28403b _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x55d26457c75c std::function<>::operator()() at ??:0
I20240501 07:59:19.418053 13040 raft_consensus.cc:482] T c197ba256dc74e6d92e34799c274b4e3 P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:19.418183 13040 raft_consensus.cc:504] T c197ba256dc74e6d92e34799c274b4e3 P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } }
I20240501 07:59:19.418709 13040 leader_election.cc:288] T c197ba256dc74e6d92e34799c274b4e3 P 13d05ebdca8c4f489c597ce550dd773d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149), a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607)
I20240501 07:59:19.419256 9055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c197ba256dc74e6d92e34799c274b4e3" candidate_uuid: "13d05ebdca8c4f489c597ce550dd773d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a505e7cb133c4392bfda8a4a20874ae9" is_pre_election: true
I20240501 07:59:19.419418 9055 raft_consensus.cc:2454] T c197ba256dc74e6d92e34799c274b4e3 P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 13d05ebdca8c4f489c597ce550dd773d in term 0.
I20240501 07:59:19.419822 8859 leader_election.cc:302] T c197ba256dc74e6d92e34799c274b4e3 P 13d05ebdca8c4f489c597ce550dd773d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13d05ebdca8c4f489c597ce550dd773d, a505e7cb133c4392bfda8a4a20874ae9; no voters:
I20240501 07:59:19.420048 13040 raft_consensus.cc:2781] T c197ba256dc74e6d92e34799c274b4e3 P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:19.420104 13040 raft_consensus.cc:482] T c197ba256dc74e6d92e34799c274b4e3 P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
@ 0x7f151a271807 kudu::Thread::SuperviseThread() at ??:0
I20240501 07:59:19.420148 13040 raft_consensus.cc:3037] T c197ba256dc74e6d92e34799c274b4e3 P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:19.421622 13040 raft_consensus.cc:504] T c197ba256dc74e6d92e34799c274b4e3 P 13d05ebdca8c4f489c597ce550dd773d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } }
I20240501 07:59:19.422130 13040 leader_election.cc:288] T c197ba256dc74e6d92e34799c274b4e3 P 13d05ebdca8c4f489c597ce550dd773d [CANDIDATE]: Term 1 election: Requested vote from peers 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149), a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607)
@ 0x7f151aaaf6db start_thread at ??:0
I20240501 07:59:19.422880 9055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c197ba256dc74e6d92e34799c274b4e3" candidate_uuid: "13d05ebdca8c4f489c597ce550dd773d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a505e7cb133c4392bfda8a4a20874ae9"
I20240501 07:59:19.423018 9055 raft_consensus.cc:3037] T c197ba256dc74e6d92e34799c274b4e3 P a505e7cb133c4392bfda8a4a20874ae9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:19.424278 9055 raft_consensus.cc:2454] T c197ba256dc74e6d92e34799c274b4e3 P a505e7cb133c4392bfda8a4a20874ae9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13d05ebdca8c4f489c597ce550dd773d in term 1.
I20240501 07:59:19.424738 8859 leader_election.cc:302] T c197ba256dc74e6d92e34799c274b4e3 P 13d05ebdca8c4f489c597ce550dd773d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13d05ebdca8c4f489c597ce550dd773d, a505e7cb133c4392bfda8a4a20874ae9; no voters:
@ 0x7f1517fe771f clone at ??:0
I20240501 07:59:19.425002 13040 raft_consensus.cc:2781] T c197ba256dc74e6d92e34799c274b4e3 P 13d05ebdca8c4f489c597ce550dd773d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:19.426378 13040 raft_consensus.cc:686] T c197ba256dc74e6d92e34799c274b4e3 P 13d05ebdca8c4f489c597ce550dd773d [term 1 LEADER]: Becoming Leader. State: Replica: 13d05ebdca8c4f489c597ce550dd773d, State: Running, Role: LEADER
I20240501 07:59:19.426534 13040 consensus_queue.cc:237] T c197ba256dc74e6d92e34799c274b4e3 P 13d05ebdca8c4f489c597ce550dd773d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } }
I20240501 07:59:19.427425 6707 raft_consensus.cc:1203] T 36aaf192042f4320b2526322424dc640 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 07:59:19.430421 6170 catalog_manager.cc:5484] T c197ba256dc74e6d92e34799c274b4e3 P 13d05ebdca8c4f489c597ce550dd773d reported cstate change: term changed from 0 to 1, leader changed from <none> to 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134). New cstate: current_term: 1 leader_uuid: "13d05ebdca8c4f489c597ce550dd773d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } health_report { overall_health: UNKNOWN } } }
I20240501 07:59:19.478842 6444 raft_consensus.cc:1203] T 144ff11d2a504f7ab7a230e6822d24ca P 8315d22c56f7439a9e1fc82f153f018c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 07:59:19.493098 6444 raft_consensus.cc:1203] T e0583fa43994452ca7553cbf2309eba1 P 8315d22c56f7439a9e1fc82f153f018c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 07:59:19.522939 6707 raft_consensus.cc:1203] T d3f2250a62c6429e886753aaffcbe883 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.9->[1.10-1.10] Dedup: 1.10->[]
I20240501 07:59:19.562885 6170 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55950:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 07:59:19.564193 13418 raft_consensus.cc:482] T ad3e528eaff349a4b3b03037652b1505 P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:19.564316 13418 raft_consensus.cc:504] T ad3e528eaff349a4b3b03037652b1505 P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } }
I20240501 07:59:19.564734 13418 leader_election.cc:288] T ad3e528eaff349a4b3b03037652b1505 P 13d05ebdca8c4f489c597ce550dd773d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133:39859), a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607)
I20240501 07:59:19.565093 8794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad3e528eaff349a4b3b03037652b1505" candidate_uuid: "13d05ebdca8c4f489c597ce550dd773d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74077b61d6f54342bf9736b4a9f2eade" is_pre_election: true
I20240501 07:59:19.565253 8794 raft_consensus.cc:2454] T ad3e528eaff349a4b3b03037652b1505 P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 13d05ebdca8c4f489c597ce550dd773d in term 0.
I20240501 07:59:19.565572 8858 leader_election.cc:302] T ad3e528eaff349a4b3b03037652b1505 P 13d05ebdca8c4f489c597ce550dd773d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13d05ebdca8c4f489c597ce550dd773d, 74077b61d6f54342bf9736b4a9f2eade; no voters:
I20240501 07:59:19.565812 13418 raft_consensus.cc:2781] T ad3e528eaff349a4b3b03037652b1505 P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:19.565887 13418 raft_consensus.cc:482] T ad3e528eaff349a4b3b03037652b1505 P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:19.565932 13418 raft_consensus.cc:3037] T ad3e528eaff349a4b3b03037652b1505 P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:19.567083 13418 raft_consensus.cc:504] T ad3e528eaff349a4b3b03037652b1505 P 13d05ebdca8c4f489c597ce550dd773d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } }
I20240501 07:59:19.567449 13418 leader_election.cc:288] T ad3e528eaff349a4b3b03037652b1505 P 13d05ebdca8c4f489c597ce550dd773d [CANDIDATE]: Term 1 election: Requested vote from peers 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133:39859), a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607)
I20240501 07:59:19.567754 8794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad3e528eaff349a4b3b03037652b1505" candidate_uuid: "13d05ebdca8c4f489c597ce550dd773d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74077b61d6f54342bf9736b4a9f2eade"
I20240501 07:59:19.567875 8794 raft_consensus.cc:3037] T ad3e528eaff349a4b3b03037652b1505 P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:19.568934 8794 raft_consensus.cc:2454] T ad3e528eaff349a4b3b03037652b1505 P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13d05ebdca8c4f489c597ce550dd773d in term 1.
I20240501 07:59:19.569303 8858 leader_election.cc:302] T ad3e528eaff349a4b3b03037652b1505 P 13d05ebdca8c4f489c597ce550dd773d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13d05ebdca8c4f489c597ce550dd773d, 74077b61d6f54342bf9736b4a9f2eade; no voters:
I20240501 07:59:19.569497 13418 raft_consensus.cc:2781] T ad3e528eaff349a4b3b03037652b1505 P 13d05ebdca8c4f489c597ce550dd773d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:19.569895 13418 raft_consensus.cc:686] T ad3e528eaff349a4b3b03037652b1505 P 13d05ebdca8c4f489c597ce550dd773d [term 1 LEADER]: Becoming Leader. State: Replica: 13d05ebdca8c4f489c597ce550dd773d, State: Running, Role: LEADER
I20240501 07:59:19.570075 13418 consensus_queue.cc:237] T ad3e528eaff349a4b3b03037652b1505 P 13d05ebdca8c4f489c597ce550dd773d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } }
I20240501 07:59:19.575929 8794 raft_consensus.cc:1203] T 1c32cd00bc344ae1b161d3f9b43b53d3 P 74077b61d6f54342bf9736b4a9f2eade [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 07:59:19.576511 6164 catalog_manager.cc:5484] T ad3e528eaff349a4b3b03037652b1505 P 13d05ebdca8c4f489c597ce550dd773d reported cstate change: term changed from 0 to 1, leader changed from <none> to 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134). New cstate: current_term: 1 leader_uuid: "13d05ebdca8c4f489c597ce550dd773d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } health_report { overall_health: UNKNOWN } } }
W20240501 07:59:19.640312 6380 connection.cc:535] client connection to 127.0.99.136:46149 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 07:59:19.640483 6380 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 07:59:19.641503 8859 connection.cc:535] client connection to 127.0.99.136:46149 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 07:59:19.641662 8859 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 07:59:19.642199 6511 connection.cc:535] client connection to 127.0.99.136:46149 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 07:59:19.642321 6511 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 07:59:19.642815 6642 connection.cc:535] client connection to 127.0.99.136:46149 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 07:59:19.642928 6642 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 07:59:19.644078 8728 connection.cc:535] client connection to 127.0.99.136:46149 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 07:59:19.644197 8728 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 07:59:19.645089 6249 connection.cc:535] client connection to 127.0.99.136:46149 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 07:59:19.645218 6249 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 07:59:19.645706 6158 connection.cc:535] client connection to 127.0.99.136:46149 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 07:59:19.645783 13963 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 07:59:19.641228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.99.136:46149 (local address 127.0.99.130:48791)
0501 07:59:19.641444 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0501 07:59:19.641461 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0501 07:59:19.641573 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 07:59:19.645208 (+ 3635us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":65,"threads_started":1}
W20240501 07:59:19.645820 6158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 07:59:19.645967 6380 consensus_peers.cc:485] T 782e37a4ebb34a68b461a74115ae8703 P 8315d22c56f7439a9e1fc82f153f018c -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240501 07:59:19.646025 6380 consensus_peers.cc:485] T bf806039d8c742d8a80ad994829bdf18 P 8315d22c56f7439a9e1fc82f153f018c -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240501 07:59:19.646067 6380 consensus_peers.cc:485] T c3a9036ccfee45498bca9681518d3699 P 8315d22c56f7439a9e1fc82f153f018c -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240501 07:59:19.646767 13968 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 07:59:19.643443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.99.136:46149 (local address 127.0.99.131:58543)
0501 07:59:19.643665 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0501 07:59:19.643679 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0501 07:59:19.643767 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 07:59:19.646328 (+ 2561us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":70,"threads_started":1}
W20240501 07:59:19.647572 13969 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 07:59:19.643810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.99.136:46149 (local address 127.0.99.132:39039)
0501 07:59:19.644000 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0501 07:59:19.644013 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0501 07:59:19.644088 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 07:59:19.646990 (+ 2902us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":62,"threads_started":1}
W20240501 07:59:19.648171 13971 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 07:59:19.644760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.99.136:46149 (local address 127.0.99.133:60819)
0501 07:59:19.644979 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0501 07:59:19.644992 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0501 07:59:19.645082 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 07:59:19.647785 (+ 2703us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":66,"threads_started":1}
W20240501 07:59:19.648758 13965 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 07:59:19.642357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.99.136:46149 (local address 127.0.99.134:48187)
0501 07:59:19.642542 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0501 07:59:19.642556 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0501 07:59:19.642650 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 07:59:19.648387 (+ 5737us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":60,"threads_started":1}
W20240501 07:59:19.648885 8859 consensus_peers.cc:485] T bab84387a40d40c3b8af7f09eb462886 P 13d05ebdca8c4f489c597ce550dd773d -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240501 07:59:19.648949 8859 leader_election.cc:334] T c197ba256dc74e6d92e34799c274b4e3 P 13d05ebdca8c4f489c597ce550dd773d [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 07:59:19.649176 8859 leader_election.cc:334] T c197ba256dc74e6d92e34799c274b4e3 P 13d05ebdca8c4f489c597ce550dd773d [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 07:59:19.649266 8859 consensus_peers.cc:485] T f0c1c05110894e39b41e6b5c5894b8d8 P 13d05ebdca8c4f489c597ce550dd773d -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240501 07:59:19.649303 8859 consensus_peers.cc:485] T c197ba256dc74e6d92e34799c274b4e3 P 13d05ebdca8c4f489c597ce550dd773d -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240501 07:59:19.649338 8859 consensus_peers.cc:485] T 0864c290fafa488abefa93d41c0a2efe P 13d05ebdca8c4f489c597ce550dd773d -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240501 07:59:19.649370 8859 consensus_peers.cc:485] T e0583fa43994452ca7553cbf2309eba1 P 13d05ebdca8c4f489c597ce550dd773d -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240501 07:59:19.649405 8859 consensus_peers.cc:485] T ff9dc384dbac47dc9795617c64688943 P 13d05ebdca8c4f489c597ce550dd773d -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240501 07:59:19.649574 8859 consensus_peers.cc:485] T fb6b46072a8d46e18b53b9f8de702859 P 13d05ebdca8c4f489c597ce550dd773d -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240501 07:59:19.649932 6511 consensus_peers.cc:485] T 6e2ac8c8857d45b88d3b51147d2609aa P 3b164cbbc00f4e8eb3254ddd349a3ff9 -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240501 07:59:19.650009 6511 consensus_peers.cc:485] T 144ff11d2a504f7ab7a230e6822d24ca P 3b164cbbc00f4e8eb3254ddd349a3ff9 -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240501 07:59:19.650063 6511 consensus_peers.cc:462] T 0b24f67cf1a64debb23ee6618c820d33 P 3b164cbbc00f4e8eb3254ddd349a3ff9 -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 07:59:19.650200 8728 consensus_peers.cc:485] T ab1666f1e5204416adb435d9bbe80433 P 74077b61d6f54342bf9736b4a9f2eade -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240501 07:59:19.650271 8728 consensus_peers.cc:485] T 36aaf192042f4320b2526322424dc640 P 74077b61d6f54342bf9736b4a9f2eade -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240501 07:59:19.650303 8728 consensus_peers.cc:485] T 6dc929e2a6ae45bd8b1f297583636bb6 P 74077b61d6f54342bf9736b4a9f2eade -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240501 07:59:19.650451 6642 consensus_peers.cc:485] T 4bc915ffbbf345cbbe1195f8dd3ab874 P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240501 07:59:19.650533 6642 consensus_peers.cc:485] T 404bb81b9c404dd288d2f8160a4273da P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240501 07:59:19.658401 6249 consensus_peers.cc:485] T 2e540f389b2a4aef93a020f549d11b67 P 8b8b3ad3aa8f4053b2167dd550f2086f -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 07:59:19.658500 6249 consensus_peers.cc:485] T 5ee738571af740c1b494d4f17413d1d8 P 8b8b3ad3aa8f4053b2167dd550f2086f -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 07:59:19.661110 6158 catalog_manager.cc:4573] TS 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): CreateTablet RPC failed for tablet 7dbbfc5d5848486f983cffdd2f36cc91: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111)
I20240501 07:59:19.671972 6707 raft_consensus.cc:1203] T e5d564b9558946f585ca7a7a3b457a07 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 07:59:19.679023 13218 consensus_queue.cc:1035] T f3375300a7914fa599fe0d3a8f1f9c18 P 74077b61d6f54342bf9736b4a9f2eade [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 07:59:19.710211 8728 connection.cc:535] client connection to 127.0.99.135:42607 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 07:59:19.710335 6511 connection.cc:535] client connection to 127.0.99.135:42607 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 07:59:19.710438 6249 connection.cc:535] client connection to 127.0.99.135:42607 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 07:59:19.710537 6642 connection.cc:535] client connection to 127.0.99.135:42607 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 07:59:19.711670 8859 connection.cc:535] client connection to 127.0.99.135:42607 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 07:59:19.712193 13985 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 07:59:19.711029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.99.135:42607 (local address 127.0.99.129:47247)
0501 07:59:19.711234 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0501 07:59:19.711250 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0501 07:59:19.711347 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 07:59:19.711600 (+ 253us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":120}
W20240501 07:59:19.712352 6249 consensus_peers.cc:485] T 0fc1ce00a2024e4a9c19abcc7d181efd P 8b8b3ad3aa8f4053b2167dd550f2086f -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240501 07:59:19.712414 6249 consensus_peers.cc:485] T 6c3c0d6fe41f4ae196ae29beb9b68125 P 8b8b3ad3aa8f4053b2167dd550f2086f -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240501 07:59:19.712693 13969 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 07:59:19.711111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.99.135:42607 (local address 127.0.99.132:54149)
0501 07:59:19.711297 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0501 07:59:19.711313 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0501 07:59:19.711402 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 07:59:19.712600 (+ 1198us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":114}
W20240501 07:59:19.712874 6642 consensus_peers.cc:462] T a29720541b7b4a79aaa810592ef7c4b1 P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 07:59:19.712934 6642 consensus_peers.cc:485] T 8454f24b0c28451b8ee51be39ca3e07f P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240501 07:59:19.713306 6380 connection.cc:535] client connection to 127.0.99.135:42607 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 07:59:19.713594 8859 leader_election.cc:334] T ad3e528eaff349a4b3b03037652b1505 P 13d05ebdca8c4f489c597ce550dd773d [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111)
W20240501 07:59:19.713747 8859 consensus_peers.cc:485] T e0583fa43994452ca7553cbf2309eba1 P 13d05ebdca8c4f489c597ce550dd773d -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 07:59:19.713804 8859 consensus_peers.cc:485] T c197ba256dc74e6d92e34799c274b4e3 P 13d05ebdca8c4f489c597ce550dd773d -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 07:59:19.713840 8859 leader_election.cc:334] T ad3e528eaff349a4b3b03037652b1505 P 13d05ebdca8c4f489c597ce550dd773d [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111)
W20240501 07:59:19.714402 6380 consensus_peers.cc:485] T cfcc814f4d2d488ba745ecaf0cc6d61a P 8315d22c56f7439a9e1fc82f153f018c -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 07:59:19.714485 6380 consensus_peers.cc:485] T bf806039d8c742d8a80ad994829bdf18 P 8315d22c56f7439a9e1fc82f153f018c -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 07:59:19.714618 13971 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 07:59:19.711111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.99.135:42607 (local address 127.0.99.133:57607)
0501 07:59:19.711389 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0501 07:59:19.711401 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0501 07:59:19.711478 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 07:59:19.714512 (+ 3034us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":206}
W20240501 07:59:19.714728 8728 consensus_peers.cc:485] T f3375300a7914fa599fe0d3a8f1f9c18 P 74077b61d6f54342bf9736b4a9f2eade -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240501 07:59:19.714974 13968 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 07:59:19.711380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.99.135:42607 (local address 127.0.99.131:43233)
0501 07:59:19.711584 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0501 07:59:19.711597 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0501 07:59:19.711666 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 07:59:19.714902 (+ 3236us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":117}
W20240501 07:59:19.715077 6511 consensus_peers.cc:485] T 114b6dbafa1c4c44ac42f1d2293e3125 P 3b164cbbc00f4e8eb3254ddd349a3ff9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240501 07:59:19.715144 6511 consensus_peers.cc:485] T d3f2250a62c6429e886753aaffcbe883 P 3b164cbbc00f4e8eb3254ddd349a3ff9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240501 07:59:19.716521 6249 consensus_peers.cc:485] T 1c32cd00bc344ae1b161d3f9b43b53d3 P 8b8b3ad3aa8f4053b2167dd550f2086f -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 07:59:19.719041 6249 consensus_peers.cc:485] T f68dbc1f65eb4f1b91a62e2de82a7676 P 8b8b3ad3aa8f4053b2167dd550f2086f -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 07:59:19.720705 6511 consensus_peers.cc:462] T 0b24f67cf1a64debb23ee6618c820d33 P 3b164cbbc00f4e8eb3254ddd349a3ff9 -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111)
W20240501 07:59:19.724053 6380 consensus_peers.cc:485] T 9461ad0f65e1489ca102f2558194ce8f P 8315d22c56f7439a9e1fc82f153f018c -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 07:59:19.736137 8859 consensus_peers.cc:485] T 239ee596e62d47169eeb6670dc62d59c P 13d05ebdca8c4f489c597ce550dd773d -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:19.743539 6444 raft_consensus.cc:1203] T 0864c290fafa488abefa93d41c0a2efe P 8315d22c56f7439a9e1fc82f153f018c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
W20240501 07:59:19.750068 6642 consensus_peers.cc:485] T 3b9b71d44f754353b2321e8c36dd4dfd P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:19.752812 6311 raft_consensus.cc:1203] T 0f9e2549fe294a93b3b25081a15b2028 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
W20240501 07:59:19.758132 6511 consensus_peers.cc:485] T d6c28cd91f624673b8e428f471a1dcce P 3b164cbbc00f4e8eb3254ddd349a3ff9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 07:59:19.764451 6642 consensus_peers.cc:485] T a29720541b7b4a79aaa810592ef7c4b1 P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 07:59:19.779762 6511 consensus_peers.cc:485] T d3f2250a62c6429e886753aaffcbe883 P 3b164cbbc00f4e8eb3254ddd349a3ff9 -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 07:59:19.783372 6380 consensus_peers.cc:485] T 782e37a4ebb34a68b461a74115ae8703 P 8315d22c56f7439a9e1fc82f153f018c -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 07:59:19.794991 8728 consensus_peers.cc:485] T 0f9e2549fe294a93b3b25081a15b2028 P 74077b61d6f54342bf9736b4a9f2eade -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 07:59:19.795258 6642 consensus_peers.cc:462] T a29720541b7b4a79aaa810592ef7c4b1 P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111)
W20240501 07:59:19.795838 6642 consensus_peers.cc:485] T 6daebcfc41754ec4aca939ca20588854 P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 07:59:19.802855 8728 consensus_peers.cc:485] T fcb3198c1fa5449781dc84756fe02bff P 74077b61d6f54342bf9736b4a9f2eade -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 07:59:19.820618 6511 consensus_peers.cc:485] T 3745acee2b5543b697a3c8f45d2618bb P 3b164cbbc00f4e8eb3254ddd349a3ff9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 07:59:19.826031 6380 consensus_peers.cc:485] T a756205204d146ce82a2799e30582f57 P 8315d22c56f7439a9e1fc82f153f018c -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:19.851672 13396 raft_consensus.cc:1050] T 36aaf192042f4320b2526322424dc640 P 74077b61d6f54342bf9736b4a9f2eade: attempting to promote NON_VOTER 1f9cb72be98941b5a8e504cbe34132b9 to VOTER
I20240501 07:59:19.852149 13396 consensus_queue.cc:237] T 36aaf192042f4320b2526322424dc640 P 74077b61d6f54342bf9736b4a9f2eade [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { replace: true } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { promote: false } }
I20240501 07:59:19.854008 8924 raft_consensus.cc:1261] T 36aaf192042f4320b2526322424dc640 P 13d05ebdca8c4f489c597ce550dd773d [term 1 FOLLOWER]: Refusing update from remote peer 74077b61d6f54342bf9736b4a9f2eade: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 07:59:19.854009 6707 raft_consensus.cc:1261] T 36aaf192042f4320b2526322424dc640 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 LEARNER]: Refusing update from remote peer 74077b61d6f54342bf9736b4a9f2eade: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 07:59:19.854847 13538 consensus_queue.cc:1035] T 36aaf192042f4320b2526322424dc640 P 74077b61d6f54342bf9736b4a9f2eade [LEADER]: Connected to new peer: Peer: permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 07:59:19.857266 13538 consensus_queue.cc:1035] T 36aaf192042f4320b2526322424dc640 P 74077b61d6f54342bf9736b4a9f2eade [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20240501 07:59:19.858038 8728 consensus_peers.cc:485] T 36aaf192042f4320b2526322424dc640 P 74077b61d6f54342bf9736b4a9f2eade -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:19.859285 13396 raft_consensus.cc:2932] T 36aaf192042f4320b2526322424dc640 P 74077b61d6f54342bf9736b4a9f2eade [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { replace: true } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { promote: false } } }
I20240501 07:59:19.860687 6707 raft_consensus.cc:2932] T 36aaf192042f4320b2526322424dc640 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { replace: true } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { promote: false } } }
I20240501 07:59:19.861290 8924 raft_consensus.cc:2932] T 36aaf192042f4320b2526322424dc640 P 13d05ebdca8c4f489c597ce550dd773d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { replace: true } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { promote: false } } }
I20240501 07:59:19.863991 6168 catalog_manager.cc:5484] T 36aaf192042f4320b2526322424dc640 P 1f9cb72be98941b5a8e504cbe34132b9 reported cstate change: config changed from index 2 to 3, 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "74077b61d6f54342bf9736b4a9f2eade" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { replace: true } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { promote: false } } }
I20240501 07:59:19.882593 6556 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 07:59:19.886355 8757 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 07:59:19.888453 8905 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 07:59:19.889211 6687 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 07:59:19.904824 6295 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 07:59:19.913589 6424 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
/home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/tools/rebalancer_tool-test.cc:1203: Failure
Value of: s.ok()
Actual: false
Expected: true
Runtime error: /tmp/dist-test-task7tbgEt/build/debug/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
Statistic | Value
-----------------------+-----------
Minimum Replica Count | 0
Maximum Replica Count | 46
Average Replica Count | 22.875000
Per-table replica distribution summary:
Replica Skew | Value
--------------+----------
Minimum | 1
Maximum | 3
Average | 2.904762
stderr: W20240501 07:59:07.375337 9242 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 07:59:07.375599 9242 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 07:59:07.379204 9242 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 07:59:07.882261 9242 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240501 07:59:08.127499 9242 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 9ec2540a0f4b49ada014e0c68ec3b2a0: could not find any suitable replica to move from server 83f20e84af4c40d09a4804b6b26e697f to server 1f9cb72be98941b5a8e504cbe34132b9
W20240501 07:59:08.127717 9242 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table c4f46595186f4427aa00d6752aa5dc9a: could not find any suitable replica to move from server 8315d22c56f7439a9e1fc82f153f018c to server a505e7cb133c4392bfda8a4a20874ae9
W20240501 07:59:08.127816 9242 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ecf5cb7235314f20bbb522737ba7fd3c: could not find any suitable replica to move from server 1f9cb72be98941b5a8e504cbe34132b9 to server 13d05ebdca8c4f489c597ce550dd773d
W20240501 07:59:08.127875 9242 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 9ec2540a0f4b49ada014e0c68ec3b2a0: could not find any suitable replica to move from server 83f20e84af4c40d09a4804b6b26e697f to server 8b8b3ad3aa8f4053b2167dd550f2086f
W20240501 07:59:08.127981 9242 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6a104310158f4500bd3056343aa6eeca: could not find any suitable replica to move from server 1f9cb72be98941b5a8e504cbe34132b9 to server a505e7cb133c4392bfda8a4a20874ae9
W20240501 07:59:08.128028 9242 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 25ecaaa6eafe48ecb3f3ff353a6d1170: could not find any suitable replica to move from server 8315d22c56f7439a9e1fc82f153f018c to server 74077b61d6f54342bf9736b4a9f2eade
W20240501 07:59:08.128114 9242 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 336b9f044a794b6a8a51d76e720b0ecd: could not find any suitable replica to move from server 3b164cbbc00f4e8eb3254ddd349a3ff9 to server 13d05ebdca8c4f489c597ce550dd773d
W20240501 07:59:08.128193 9242 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 9ec2540a0f4b49ada014e0c68ec3b2a0: could not find any suitable replica to move from server 13d05ebdca8c4f489c597ce550dd773d to server 8315d22c56f7439a9e1fc82f153f018c
W20240501 07:59:08.128235 9242 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table bdae856e60144776aae709dddb837795: could not find any suitable replica to move from server 8315d22c56f7439a9e1fc82f153f018c to server 13d05ebdca8c4f489c597ce550dd773d
W20240501 07:59:08.128288 9242 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 09bfb3350ac042c892c1ef5bc6adf5e2: could not find any suitable replica to move from server 8315d22c56f7439a9e1fc82f153f018c to server 74077b61d6f54342bf9736b4a9f2eade
W20240501 07:59:08.128329 9242 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table d0e1f982440849beb494081bea9cb211: could not find any suitable replica to move from server 1f9cb72be98941b5a8e504cbe34132b9 to server a505e7cb133c4392bfda8a4a20874ae9
W20240501 07:59:08.128360 9242 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 25ecaaa6eafe48ecb3f3ff353a6d1170: could not find any suitable replica to move from server 1f9cb72be98941b5a8e504cbe34132b9 to server 13d05ebdca8c4f489c597ce550dd773d
W20240501 07:59:08.128401 9242 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a8027b31a8b34320800f512d75105639: could not find any suitable replica to move from server 8b8b3ad3aa8f4053b2167dd550f2086f to server 83f20e84af4c40d09a4804b6b26e697f
W20240501 07:59:08.128434 9242 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 886e8147658c4cffb480ca3fb821c563: could not find any suitable replica to move from server 1f9cb72be98941b5a8e504cbe34132b9 to server 74077b61d6f54342bf9736b4a9f2eade
W20240501 07:59:08.128463 9242 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 4de37f887eb14e2dbe5d1609df1e538f: could not find any suitable replica to move from server 3b164cbbc00f4e8eb3254ddd349a3ff9 to server a505e7cb133c4392bfda8a4a20874ae9
W20240501 07:59:08.128491 9242 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 7b24d2d13db34edb849642d7d3eef300: could not find any suitable replica to move from server 8315d22c56f7439a9e1fc82f153f018c to server a505e7cb133c4392bfda8a4a20874ae9
W20240501 07:59:08.128532 9242 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table e3d01900e54c4eb1920ccd2ff7a4f5db: could not find any suitable replica to move from server 3b164cbbc00f4e8eb3254ddd349a3ff9 to server 13d05ebdca8c4f489c597ce550dd773d
W20240501 07:59:08.128576 9242 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table eb6765f7643442ff82fde7f790b2aaed: could not find any suitable replica to move from server 8b8b3ad3aa8f4053b2167dd550f2086f to server 13d05ebdca8c4f489c597ce550dd773d
W20240501 07:59:08.128607 9242 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3e062fd8d58941efbd3d190c28bb0059: could not find any suitable replica to move from server 8b8b3ad3aa8f4053b2167dd550f2086f to server 83f20e84af4c40d09a4804b6b26e697f
W20240501 07:59:08.128635 9242 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 35f40c7e6b1e4d799576a7d98130d848: could not find any suitable replica to move from server 3b164cbbc00f4e8eb3254ddd349a3ff9 to server 74077b61d6f54342bf9736b4a9f2eade
W20240501 07:59:08.128660 9242 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 85a512f46553493eaf7ce4a1eecac55f: could not find any suitable replica to move from server 8b8b3ad3aa8f4053b2167dd550f2086f to server 74077b61d6f54342bf9736b4a9f2eade
W20240501 07:59:08.128700 9242 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f041050203204d0690b4b8f368f834b2: could not find any suitable replica to move from server 3b164cbbc00f4e8eb3254ddd349a3ff9 to server a505e7cb133c4392bfda8a4a20874ae9
W20240501 07:59:08.128731 9242 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 8e242b97203b46d0840cb3b6a649e5a7: could not find any suitable replica to move from server 8315d22c56f7439a9e1fc82f153f018c to server 83f20e84af4c40d09a4804b6b26e697f
W20240501 07:59:08.128760 9242 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 09bfb3350ac042c892c1ef5bc6adf5e2: could not find any suitable replica to move from server 1f9cb72be98941b5a8e504cbe34132b9 to server a505e7cb133c4392bfda8a4a20874ae9
W20240501 07:59:08.128787 9242 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6a104310158f4500bd3056343aa6eeca: could not find any suitable replica to move from server 8b8b3ad3aa8f4053b2167dd550f2086f to server 13d05ebdca8c4f489c597ce550dd773d
W20240501 07:59:08.128816 9242 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table c4f46595186f4427aa00d6752aa5dc9a: could not find any suitable replica to move from server 8315d22c56f7439a9e1fc82f153f018c to server 83f20e84af4c40d09a4804b6b26e697f
W20240501 07:59:08.128849 9242 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table d0e1f982440849beb494081bea9cb211: could not find any suitable replica to move from server 8b8b3ad3aa8f4053b2167dd550f2086f to server a505e7cb133c4392bfda8a4a20874ae9
W20240501 07:59:08.128881 9242 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a8027b31a8b34320800f512d75105639: could not find any suitable replica to move from server 1f9cb72be98941b5a8e504cbe34132b9 to server 74077b61d6f54342bf9736b4a9f2eade
W20240501 07:59:08.128912 9242 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 886e8147658c4cffb480ca3fb821c563: could not find any suitable replica to move from server 8315d22c56f7439a9e1fc82f153f018c to server 83f20e84af4c40d09a4804b6b26e697f
W20240501 07:59:08.128947 9242 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3e062fd8d58941efbd3d190c28bb0059: could not find any suitable replica to move from server 3b164cbbc00f4e8eb3254ddd349a3ff9 to server 13d05ebdca8c4f489c597ce550dd773d
I20240501 07:59:08.145838 9242 rebalancer_tool.cc:1074] tablet 26713549654249b09f820a31ee19e2b7: '8b8b3ad3aa8f4053b2167dd550f2086f' -> 'a505e7cb133c4392bfda8a4a20874ae9' move scheduled
I20240501 07:59:08.164042 9242 rebalancer_tool.cc:1074] tablet 271e3c5cf9934eaead7fa1ba942084be: '3b164cbbc00f4e8eb3254ddd349a3ff9' -> '74077b61d6f54342bf9736b4a9f2eade' move scheduled
I20240501 07:59:08.179482 9242 rebalancer_tool.cc:1074] tablet 7d63a75f2194484f8a59709a209f00ea: '1f9cb72be98941b5a8e504cbe34132b9' -> '13d05ebdca8c4f489c597ce550dd773d' move scheduled
I20240501 07:59:08.197587 9242 rebalancer_tool.cc:1074] tablet d3f2250a62c6429e886753aaffcbe883: '8315d22c56f7439a9e1fc82f153f018c' -> '83f20e84af4c40d09a4804b6b26e697f' move scheduled
I20240501 07:59:08.223235 9242 rebalancer_tool.cc:1074] tablet 810a2791eb04415dae47e85db87352cb: '8b8b3ad3aa8f4053b2167dd550f2086f' -> 'a505e7cb133c4392bfda8a4a20874ae9' move scheduled
I20240501 07:59:08.240862 9242 rebalancer_tool.cc:1074] tablet 6dc929e2a6ae45bd8b1f297583636bb6: '3b164cbbc00f4e8eb3254ddd349a3ff9' -> '74077b61d6f54342bf9736b4a9f2eade' move scheduled
I20240501 07:59:08.263146 9242 rebalancer_tool.cc:1074] tablet ec2d3ccf0937430f947f3e2bde0506c5: '1f9cb72be98941b5a8e504cbe34132b9' -> '13d05ebdca8c4f489c597ce550dd773d' move scheduled
I20240501 07:59:08.292380 9242 rebalancer_tool.cc:1074] tablet 404bb81b9c404dd288d2f8160a4273da: '8315d22c56f7439a9e1fc82f153f018c' -> '83f20e84af4c40d09a4804b6b26e697f' move scheduled
I20240501 07:59:08.306526 9242 rebalancer_tool.cc:1074] tablet 3b9b71d44f754353b2321e8c36dd4dfd: '8b8b3ad3aa8f4053b2167dd550f2086f' -> 'a505e7cb133c4392bfda8a4a20874ae9' move scheduled
I20240501 07:59:08.323930 9242 rebalancer_tool.cc:1074] tablet ff9dc384dbac47dc9795617c64688943: '3b164cbbc00f4e8eb3254ddd349a3ff9' -> '13d05ebdca8c4f489c597ce550dd773d' move scheduled
I20240501 07:59:08.338829 9242 rebalancer_tool.cc:1074] tablet 5ee738571af740c1b494d4f17413d1d8: '1f9cb72be98941b5a8e504cbe34132b9' -> '74077b61d6f54342bf9736b4a9f2eade' move scheduled
I20240501 07:59:08.356166 9242 rebalancer_tool.cc:1074] tablet 6e2ac8c8857d45b88d3b51147d2609aa: '8315d22c56f7439a9e1fc82f153f018c' -> '83f20e84af4c40d09a4804b6b26e697f' move scheduled
I20240501 07:59:08.380630 9242 rebalancer_tool.cc:1074] tablet 187940ba79f546b1ad79e7ec1de76442: '8b8b3ad3aa8f4053b2167dd550f2086f' -> '13d05ebdca8c4f489c597ce550dd773d' move scheduled
I20240501 07:59:08.406080 9242 rebalancer_tool.cc:1074] tablet 782e37a4ebb34a68b461a74115ae8703: '3b164cbbc00f4e8eb3254ddd349a3ff9' -> 'a505e7cb133c4392bfda8a4a20874ae9' move scheduled
I20240501 07:59:08.424052 9242 rebalancer_tool.cc:1074] tablet c3a9036ccfee45498bca9681518d3699: '1f9cb72be98941b5a8e504cbe34132b9' -> '74077b61d6f54342bf9736b4a9f2eade' move scheduled
I20240501 07:59:08.439222 9242 rebalancer_tool.cc:1074] tablet bab34f083c8045e4a6df373db5ebd54b: '8315d22c56f7439a9e1fc82f153f018c' -> '13d05ebdca8c4f489c597ce550dd773d' move scheduled
I20240501 07:59:08.462544 9242 rebalancer_tool.cc:1074] tablet 7d51b1940b594c79a7d13d7026b76da8: '8b8b3ad3aa8f4053b2167dd550f2086f' -> '83f20e84af4c40d09a4804b6b26e697f' move scheduled
I20240501 07:59:08.481451 9242 rebalancer_tool.cc:1074] tablet 8454f24b0c28451b8ee51be39ca3e07f: '3b164cbbc00f4e8eb3254ddd349a3ff9' -> 'a505e7cb133c4392bfda8a4a20874ae9' move scheduled
I20240501 07:59:08.524713 9242 rebalancer_tool.cc:1074] tablet b7d57b7fd5234f50afbc859bc997f3e1: '1f9cb72be98941b5a8e504cbe34132b9' -> '74077b61d6f54342bf9736b4a9f2eade' move scheduled
I20240501 07:59:08.589602 9242 rebalancer_tool.cc:1123] tablet 7d63a75f2194484f8a59709a209f00ea: '1f9cb72be98941b5a8e504cbe34132b9' -> '13d05ebdca8c4f489c597ce550dd773d' move is abandoned: Not found: 7d63a75f2194484f8a59709a209f00ea: tablet not found
I20240501 07:59:08.693262 9242 rebalancer_tool.cc:756] re-synchronizing cluster state
E20240501 07:59:08.778666 9823 ksck_remote.cc:630] unable to open table rebalance_test_table_4: Not found: the table does not exist: table_name: "rebalance_test_table_4"
E20240501 07:59:08.800994 9833 ksck_remote.cc:630] unable to open table rebalancer_dynamic_table: Not found: the table does not exist: table_name: "rebalancer_dynamic_table"
I20240501 07:59:08.935114 9242 rebalancer_tool.cc:1123] tablet 810a2791eb04415dae47e85db87352cb: '8b8b3ad3aa8f4053b2167dd550f2086f' -> 'a505e7cb133c4392bfda8a4a20874ae9' move is abandoned: Not found: 810a2791eb04415dae47e85db87352cb: tablet not found
I20240501 07:59:09.166947 9242 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 07:59:09.291852 10042 meta_cache.cc:788] Not found: LookupRpcById { tablet: 'ec2d3ccf0937430f947f3e2bde0506c5', attempt: 1 } failed
W20240501 07:59:09.448743 9242 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table e37d9a776ed7492ba6f616e4727f982b: could not find any suitable replica to move from server a505e7cb133c4392bfda8a4a20874ae9 to server 3b164cbbc00f4e8eb3254ddd349a3ff9
W20240501 07:59:09.448884 9242 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ecf5cb7235314f20bbb522737ba7fd3c: could not find any suitable replica to move from server 1f9cb72be98941b5a8e504cbe34132b9 to server 83f20e84af4c40d09a4804b6b26e697f
W20240501 07:59:09.449009 9242 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 5e1db4e06c5e470a9d2f8a20dd23c585: could not find any suitable replica to move from server 8315d22c56f7439a9e1fc82f153f018c to server 8b8b3ad3aa8f4053b2167dd550f2086f
W20240501 07:59:09.449081 9242 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 35f40c7e6b1e4d799576a7d98130d848: could not find any suitable replica to move from server 1f9cb72be98941b5a8e504cbe34132b9 to server 74077b61d6f54342bf9736b4a9f2eade
W20240501 07:59:09.449199 9242 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 336b9f044a794b6a8a51d76e720b0ecd: could not find any suitable replica to move from server 1f9cb72be98941b5a8e504cbe34132b9 to server a505e7cb133c4392bfda8a4a20874ae9
W20240501 07:59:09.449249 9242 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 85a512f46553493eaf7ce4a1eecac55f: could not find any suitable replica to move from server 3b164cbbc00f4e8eb3254ddd349a3ff9 to server 74077b61d6f54342bf9736b4a9f2eade
W20240501 07:59:09.449308 9242 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table eb6765f7643442ff82fde7f790b2aaed: could not find any suitable replica to move from server 8b8b3ad3aa8f4053b2167dd550f2086f to server 13d05ebdca8c4f489c597ce550dd773d
W20240501 07:59:09.449348 9242 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 8e242b97203b46d0840cb3b6a649e5a7: could not find any suitable replica to move from server 8315d22c56f7439a9e1fc82f153f018c to server 83f20e84af4c40d09a4804b6b26e697f
W20240501 07:59:09.449411 9242 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table d0e1f982440849beb494081bea9cb211: could not find any suitable replica to move from server 1f9cb72be98941b5a8e504cbe34132b9 to server 83f20e84af4c40d09a4804b6b26e697f
W20240501 07:59:09.449451 9242 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 4de37f887eb14e2dbe5d1609df1e538f: could not find any suitable replica to move from server 3b164cbbc00f4e8eb3254ddd349a3ff9 to server 74077b61d6f54342bf9736b4a9f2eade
W20240501 07:59:09.449496 9242 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 7b24d2d13db34edb849642d7d3eef300: could not find any suitable replica to move from server 8b8b3ad3aa8f4053b2167dd550f2086f to server 74077b61d6f54342bf9736b4a9f2eade
W20240501 07:59:09.449537 9242 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 886e8147658c4cffb480ca3fb821c563: could not find any suitable replica to move from server 8315d22c56f7439a9e1fc82f153f018c to server 74077b61d6f54342bf9736b4a9f2eade
W20240501 07:59:09.449631 9242 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table c4f46595186f4427aa00d6752aa5dc9a: could not find any suitable replica to move from server 1f9cb72be98941b5a8e504cbe34132b9 to server 83f20e84af4c40d09a4804b6b26e697f
W20240501 07:59:09.449668 9242 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f041050203204d0690b4b8f368f834b2: could not find any suitable replica to move from server 3b164cbbc00f4e8eb3254ddd349a3ff9 to server 83f20e84af4c40d09a4804b6b26e697f
W20240501 07:59:09.449724 9242 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3e062fd8d58941efbd3d190c28bb0059: could not find any suitable replica to move from server 8b8b3ad3aa8f4053b2167dd550f2086f to server a505e7cb133c4392bfda8a4a20874ae9
W20240501 07:59:09.449766 9242 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6a104310158f4500bd3056343aa6eeca: could not find any suitable replica to move from server 1f9cb72be98941b5a8e504cbe34132b9 to server 83f20e84af4c40d09a4804b6b26e697f
W20240501 07:59:09.449805 9242 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 25ecaaa6eafe48ecb3f3ff353a6d1170: could not find any suitable replica to move from server 8315d22c56f7439a9e1fc82f153f018c to server a505e7cb133c4392bfda8a4a20874ae9
W20240501 07:59:09.449855 9242 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table bdae856e60144776aae709dddb837795: could not find any suitable replica to move from server 3b164cbbc00f4e8eb3254ddd349a3ff9 to server 83f20e84af4c40d09a4804b6b26e697f
W20240501 07:59:09.451043 9242 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 4de37f887eb14e2dbe5d1609df1e538f: could not find any suitable replica to move from server 1f9cb72be98941b5a8e504cbe34132b9 to server a505e7cb133c4392bfda8a4a20874ae9
W20240501 07:59:09.451084 9242 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 8e242b97203b46d0840cb3b6a649e5a7: could not find any suitable replica to move from server 8b8b3ad3aa8f4053b2167dd550f2086f to server 13d05ebdca8c4f489c597ce550dd773d
W20240501 07:59:09.451124 9242 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ecf5cb7235314f20bbb522737ba7fd3c: could not find any suitable replica to move from server 8315d22c56f7439a9e1fc82f153f018c to server a505e7cb133c4392bfda8a4a20874ae9
W20240501 07:59:09.451175 9242 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 886e8147658c4cffb480ca3fb821c563: could not find any suitable replica to move from server 1f9cb72be98941b5a8e504cbe34132b9 to server a505e7cb133c4392bfda8a4a20874ae9
I20240501 07:59:09.476038 9242 rebalancer_tool.cc:1074] tablet 3745acee2b5543b697a3c8f45d2618bb: '8b8b3ad3aa8f4053b2167dd550f2086f' -> 'a505e7cb133c4392bfda8a4a20874ae9' move scheduled
I20240501 07:59:09.502089 9242 rebalancer_tool.cc:1074] tablet 72feac2a993141168ac9399d9361ef17: '8315d22c56f7439a9e1fc82f153f018c' -> '13d05ebdca8c4f489c597ce550dd773d' move scheduled
I20240501 07:59:09.520249 9242 rebalancer_tool.cc:1074] tablet f9df6f3df2b94e5eb97dbff04dbe2ae1: '1f9cb72be98941b5a8e504cbe34132b9' -> '83f20e84af4c40d09a4804b6b26e697f' move scheduled
I20240501 07:59:09.571337 9242 rebalancer_tool.cc:1123] tablet 26713549654249b09f820a31ee19e2b7: '8b8b3ad3aa8f4053b2167dd550f2086f' -> 'a505e7cb133c4392bfda8a4a20874ae9' move is abandoned: Not found: 26713549654249b09f820a31ee19e2b7: tablet not found
I20240501 07:59:09.572026 9242 rebalancer_tool.cc:1123] tablet ec2d3ccf0937430f947f3e2bde0506c5: '1f9cb72be98941b5a8e504cbe34132b9' -> '13d05ebdca8c4f489c597ce550dd773d' move is abandoned: Not found: ec2d3ccf0937430f947f3e2bde0506c5: tablet not found
I20240501 07:59:09.653658 9242 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 07:59:09.857614 10224 meta_cache.cc:1015] Not found: LookupRpc { table: 'rebalance_test_table_8', partition-key: (<start>), attempt: 1 } failed: table rebalance_test_table_8 was deleted: Table deleted at 2024-05-01 07:59:09 UTC
W20240501 07:59:10.202766 9242 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table c4f46595186f4427aa00d6752aa5dc9a: could not find any suitable replica to move from server 8315d22c56f7439a9e1fc82f153f018c to server a505e7cb133c4392bfda8a4a20874ae9
W20240501 07:59:10.202895 9242 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 7b24d2d13db34edb849642d7d3eef300: could not find any suitable replica to move from server 8b8b3ad3aa8f4053b2167dd550f2086f to server a505e7cb133c4392bfda8a4a20874ae9
W20240501 07:59:10.203025 9242 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 886e8147658c4cffb480ca3fb821c563: could not find any suitable replica to move from server 1f9cb72be98941b5a8e504cbe34132b9 to server 74077b61d6f54342bf9736b4a9f2eade
W20240501 07:59:10.203094 9242 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 5e1db4e06c5e470a9d2f8a20dd23c585: could not find any suitable replica to move from server 8315d22c56f7439a9e1fc82f153f018c to server 8b8b3ad3aa8f4053b2167dd550f2086f
W20240501 07:59:10.203197 9242 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 336b9f044a794b6a8a51d76e720b0ecd: could not find any suitable replica to move from server 3b164cbbc00f4e8eb3254ddd349a3ff9 to server 74077b61d6f54342bf9736b4a9f2eade
W20240501 07:59:10.203253 9242 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6a104310158f4500bd3056343aa6eeca: could not find any suitable replica to move from server 1f9cb72be98941b5a8e504cbe34132b9 to server a505e7cb133c4392bfda8a4a20874ae9
W20240501 07:59:10.203315 9242 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ecf5cb7235314f20bbb522737ba7fd3c: could not find any suitable replica to move from server 1f9cb72be98941b5a8e504cbe34132b9 to server 13d05ebdca8c4f489c597ce550dd773d
W20240501 07:59:10.203377 9242 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table d0e1f982440849beb494081bea9cb211: could not find any suitable replica to move from server 8b8b3ad3aa8f4053b2167dd550f2086f to server 74077b61d6f54342bf9736b4a9f2eade
W20240501 07:59:10.203416 9242 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table eb6765f7643442ff82fde7f790b2aaed: could not find any suitable replica to move from server 8b8b3ad3aa8f4053b2167dd550f2086f to server 83f20e84af4c40d09a4804b6b26e697f
W20240501 07:59:10.203460 9242 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 8e242b97203b46d0840cb3b6a649e5a7: could not find any suitable replica to move from server 8b8b3ad3aa8f4053b2167dd550f2086f to server 13d05ebdca8c4f489c597ce550dd773d
W20240501 07:59:10.203505 9242 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3e062fd8d58941efbd3d190c28bb0059: could not find any suitable replica to move from server 8315d22c56f7439a9e1fc82f153f018c to server 13d05ebdca8c4f489c597ce550dd773d
W20240501 07:59:10.203727 9242 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 4de37f887eb14e2dbe5d1609df1e538f: could not find any suitable replica to move from server 1f9cb72be98941b5a8e504cbe34132b9 to server 74077b61d6f54342bf9736b4a9f2eade
W20240501 07:59:10.203789 9242 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 35f40c7e6b1e4d799576a7d98130d848: could not find any suitable replica to move from server 8b8b3ad3aa8f4053b2167dd550f2086f to server 13d05ebdca8c4f489c597ce550dd773d
W20240501 07:59:10.203869 9242 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f041050203204d0690b4b8f368f834b2: could not find any suitable replica to move from server 1f9cb72be98941b5a8e504cbe34132b9 to server 83f20e84af4c40d09a4804b6b26e697f
W20240501 07:59:10.203907 9242 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 25ecaaa6eafe48ecb3f3ff353a6d1170: could not find any suitable replica to move from server 8315d22c56f7439a9e1fc82f153f018c to server 83f20e84af4c40d09a4804b6b26e697f
W20240501 07:59:10.204002 9242 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table bdae856e60144776aae709dddb837795: could not find any suitable replica to move from server 3b164cbbc00f4e8eb3254ddd349a3ff9 to server 13d05ebdca8c4f489c597ce550dd773d
W20240501 07:59:10.204039 9242 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table c4f46595186f4427aa00d6752aa5dc9a: could not find any suitable replica to move from server 1f9cb72be98941b5a8e504cbe34132b9 to server 74077b61d6f54342bf9736b4a9f2eade
W20240501 07:59:10.204078 9242 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ecf5cb7235314f20bbb522737ba7fd3c: could not find any suitable replica to move from server 3b164cbbc00f4e8eb3254ddd349a3ff9 to server a505e7cb133c4392bfda8a4a20874ae9
W20240501 07:59:10.204115 9242 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table eb6765f7643442ff82fde7f790b2aaed: could not find any suitable replica to move from server 8315d22c56f7439a9e1fc82f153f018c to server 13d05ebdca8c4f489c597ce550dd773d
I20240501 07:59:10.227388 9242 rebalancer_tool.cc:1074] tablet adf207c019994dd49671bab4145fc268: '8b8b3ad3aa8f4053b2167dd550f2086f' -> 'a505e7cb133c4392bfda8a4a20874ae9' move scheduled
I20240501 07:59:10.279321 9242 rebalancer_tool.cc:1136] tablet 8454f24b0c28451b8ee51be39ca3e07f: '3b164cbbc00f4e8eb3254ddd349a3ff9' -> 'a505e7cb133c4392bfda8a4a20874ae9' move completed: OK
I20240501 07:59:10.285286 9242 rebalancer_tool.cc:1136] tablet b7d57b7fd5234f50afbc859bc997f3e1: '1f9cb72be98941b5a8e504cbe34132b9' -> '74077b61d6f54342bf9736b4a9f2eade' move completed: OK
I20240501 07:59:10.303457 9242 rebalancer_tool.cc:1136] tablet 782e37a4ebb34a68b461a74115ae8703: '3b164cbbc00f4e8eb3254ddd349a3ff9' -> 'a505e7cb133c4392bfda8a4a20874ae9' move completed: OK
I20240501 07:59:10.314108 9242 rebalancer_tool.cc:1136] tablet d3f2250a62c6429e886753aaffcbe883: '8315d22c56f7439a9e1fc82f153f018c' -> '83f20e84af4c40d09a4804b6b26e697f' move completed: OK
I20240501 07:59:10.315093 9242 rebalancer_tool.cc:1123] tablet 7d51b1940b594c79a7d13d7026b76da8: '8b8b3ad3aa8f4053b2167dd550f2086f' -> '83f20e84af4c40d09a4804b6b26e697f' move is abandoned: Not found: 7d51b1940b594c79a7d13d7026b76da8: tablet not found
I20240501 07:59:10.321241 9242 rebalancer_tool.cc:1136] tablet 3b9b71d44f754353b2321e8c36dd4dfd: '8b8b3ad3aa8f4053b2167dd550f2086f' -> 'a505e7cb133c4392bfda8a4a20874ae9' move completed: OK
I20240501 07:59:10.326946 9242 rebalancer_tool.cc:1136] tablet 271e3c5cf9934eaead7fa1ba942084be: '3b164cbbc00f4e8eb3254ddd349a3ff9' -> '74077b61d6f54342bf9736b4a9f2eade' move completed: OK
I20240501 07:59:10.333325 9242 rebalancer_tool.cc:1136] tablet 6e2ac8c8857d45b88d3b51147d2609aa: '8315d22c56f7439a9e1fc82f153f018c' -> '83f20e84af4c40d09a4804b6b26e697f' move completed: OK
I20240501 07:59:10.346007 9242 rebalancer_tool.cc:1136] tablet 187940ba79f546b1ad79e7ec1de76442: '8b8b3ad3aa8f4053b2167dd550f2086f' -> '13d05ebdca8c4f489c597ce550dd773d' move completed: OK
I20240501 07:59:10.379554 9242 rebalancer_tool.cc:1136] tablet 6dc929e2a6ae45bd8b1f297583636bb6: '3b164cbbc00f4e8eb3254ddd349a3ff9' -> '74077b61d6f54342bf9736b4a9f2eade' move completed: OK
I20240501 07:59:10.385221 9242 rebalancer_tool.cc:1136] tablet 404bb81b9c404dd288d2f8160a4273da: '8315d22c56f7439a9e1fc82f153f018c' -> '83f20e84af4c40d09a4804b6b26e697f' move completed: OK
I20240501 07:59:10.390765 9242 rebalancer_tool.cc:1136] tablet 5ee738571af740c1b494d4f17413d1d8: '1f9cb72be98941b5a8e504cbe34132b9' -> '74077b61d6f54342bf9736b4a9f2eade' move completed: OK
I20240501 07:59:10.421607 9242 rebalancer_tool.cc:1074] tablet 2398430a23094711a13a86cc88e27a78: '3b164cbbc00f4e8eb3254ddd349a3ff9' -> '74077b61d6f54342bf9736b4a9f2eade' move scheduled
I20240501 07:59:10.447250 9242 rebalancer_tool.cc:1074] tablet 33b3834db0cd4407ae7c26e044459e87: '8315d22c56f7439a9e1fc82f153f018c' -> 'a505e7cb133c4392bfda8a4a20874ae9' move scheduled
I20240501 07:59:10.467954 9242 rebalancer_tool.cc:1074] tablet 6292ff0c7e624dd7b24715b83a30fff8: '83f20e84af4c40d09a4804b6b26e697f' -> '1f9cb72be98941b5a8e504cbe34132b9' move scheduled
I20240501 07:59:10.491266 9242 rebalancer_tool.cc:1074] tablet 0b680bccf110443e9495942863364ff9: '3b164cbbc00f4e8eb3254ddd349a3ff9' -> '74077b61d6f54342bf9736b4a9f2eade' move scheduled
I20240501 07:59:10.510632 9242 rebalancer_tool.cc:1074] tablet b284ed2cb0264ced99a4ffd90983d902: '8b8b3ad3aa8f4053b2167dd550f2086f' -> '13d05ebdca8c4f489c597ce550dd773d' move scheduled
I20240501 07:59:10.526348 9242 rebalancer_tool.cc:1074] tablet f68dbc1f65eb4f1b91a62e2de82a7676: '8315d22c56f7439a9e1fc82f153f018c' -> '83f20e84af4c40d09a4804b6b26e697f' move scheduled
I20240501 07:59:10.548565 9242 rebalancer_tool.cc:1074] tablet 6c3c0d6fe41f4ae196ae29beb9b68125: '1f9cb72be98941b5a8e504cbe34132b9' -> 'a505e7cb133c4392bfda8a4a20874ae9' move scheduled
I20240501 07:59:10.563160 9242 rebalancer_tool.cc:1074] tablet e92d6aeeb3df4ea3852c6ebbed0e07c5: '3b164cbbc00f4e8eb3254ddd349a3ff9' -> '13d05ebdca8c4f489c597ce550dd773d' move scheduled
I20240501 07:59:10.579295 9242 rebalancer_tool.cc:1074] tablet fb6b46072a8d46e18b53b9f8de702859: '74077b61d6f54342bf9736b4a9f2eade' -> '8b8b3ad3aa8f4053b2167dd550f2086f' move scheduled
I20240501 07:59:10.605177 9242 rebalancer_tool.cc:1074] tablet 2e540f389b2a4aef93a020f549d11b67: '8315d22c56f7439a9e1fc82f153f018c' -> '83f20e84af4c40d09a4804b6b26e697f' move scheduled
I20240501 07:59:10.621574 9242 rebalancer_tool.cc:1074] tablet 114b6dbafa1c4c44ac42f1d2293e3125: '1f9cb72be98941b5a8e504cbe34132b9' -> 'a505e7cb133c4392bfda8a4a20874ae9' move scheduled
I20240501 07:59:10.731268 9242 rebalancer_tool.cc:1123] tablet f9df6f3df2b94e5eb97dbff04dbe2ae1: '1f9cb72be98941b5a8e504cbe34132b9' -> '83f20e84af4c40d09a4804b6b26e697f' move is abandoned: Not found: f9df6f3df2b94e5eb97dbff04dbe2ae1: tablet not found
I20240501 07:59:10.767383 9242 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 07:59:11.222229 9242 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 336b9f044a794b6a8a51d76e720b0ecd: could not find any suitable replica to move from server 8315d22c56f7439a9e1fc82f153f018c to server 74077b61d6f54342bf9736b4a9f2eade
W20240501 07:59:11.222359 9242 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table c4f46595186f4427aa00d6752aa5dc9a: could not find any suitable replica to move from server 1f9cb72be98941b5a8e504cbe34132b9 to server a505e7cb133c4392bfda8a4a20874ae9
W20240501 07:59:11.222427 9242 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 886e8147658c4cffb480ca3fb821c563: could not find any suitable replica to move from server 3b164cbbc00f4e8eb3254ddd349a3ff9 to server a505e7cb133c4392bfda8a4a20874ae9
W20240501 07:59:11.222496 9242 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table d0e1f982440849beb494081bea9cb211: could not find any suitable replica to move from server 1f9cb72be98941b5a8e504cbe34132b9 to server 83f20e84af4c40d09a4804b6b26e697f
W20240501 07:59:11.222543 9242 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 8e242b97203b46d0840cb3b6a649e5a7: could not find any suitable replica to move from server 3b164cbbc00f4e8eb3254ddd349a3ff9 to server 83f20e84af4c40d09a4804b6b26e697f
W20240501 07:59:11.222599 9242 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 25ecaaa6eafe48ecb3f3ff353a6d1170: could not find any suitable replica to move from server 8b8b3ad3aa8f4053b2167dd550f2086f to server 74077b61d6f54342bf9736b4a9f2eade
W20240501 07:59:11.225018 9242 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table bdae856e60144776aae709dddb837795: could not find any suitable replica to move from server 3b164cbbc00f4e8eb3254ddd349a3ff9 to server 13d05ebdca8c4f489c597ce550dd773d
W20240501 07:59:11.225122 9242 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 4de37f887eb14e2dbe5d1609df1e538f: could not find any suitable replica to move from server 8315d22c56f7439a9e1fc82f153f018c to server 74077b61d6f54342bf9736b4a9f2eade
W20240501 07:59:11.225179 9242 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6a104310158f4500bd3056343aa6eeca: could not find any suitable replica to move from server 1f9cb72be98941b5a8e504cbe34132b9 to server a505e7cb133c4392bfda8a4a20874ae9
W20240501 07:59:11.225420 9242 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 5e1db4e06c5e470a9d2f8a20dd23c585: could not find any suitable replica to move from server 8315d22c56f7439a9e1fc82f153f018c to server 13d05ebdca8c4f489c597ce550dd773d
W20240501 07:59:11.225492 9242 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 35f40c7e6b1e4d799576a7d98130d848: could not find any suitable replica to move from server 8b8b3ad3aa8f4053b2167dd550f2086f to server 74077b61d6f54342bf9736b4a9f2eade
W20240501 07:59:11.225543 9242 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table c4f46595186f4427aa00d6752aa5dc9a: could not find any suitable replica to move from server 8315d22c56f7439a9e1fc82f153f018c to server 83f20e84af4c40d09a4804b6b26e697f
I20240501 07:59:11.263952 9242 rebalancer_tool.cc:1074] tablet aa1b8716228b4fd9af95870b7a613fb2: '1f9cb72be98941b5a8e504cbe34132b9' -> '83f20e84af4c40d09a4804b6b26e697f' move scheduled
I20240501 07:59:11.287686 9242 rebalancer_tool.cc:1074] tablet 6e91683ff22243c1b3877733d74e2886: '3b164cbbc00f4e8eb3254ddd349a3ff9' -> '74077b61d6f54342bf9736b4a9f2eade' move scheduled
I20240501 07:59:11.305794 9242 rebalancer_tool.cc:1074] tablet 782e37a4ebb34a68b461a74115ae8703: '8b8b3ad3aa8f4053b2167dd550f2086f' -> '83f20e84af4c40d09a4804b6b26e697f' move scheduled
I20240501 07:59:11.344614 9242 rebalancer_tool.cc:1123] tablet 0b680bccf110443e9495942863364ff9: '3b164cbbc00f4e8eb3254ddd349a3ff9' -> '74077b61d6f54342bf9736b4a9f2eade' move is abandoned: Not found: 0b680bccf110443e9495942863364ff9: tablet not found
I20240501 07:59:11.411633 9242 rebalancer_tool.cc:1123] tablet adf207c019994dd49671bab4145fc268: '8b8b3ad3aa8f4053b2167dd550f2086f' -> 'a505e7cb133c4392bfda8a4a20874ae9' move is abandoned: Not found: adf207c019994dd49671bab4145fc268: tablet not found
I20240501 07:59:11.418888 9242 rebalancer_tool.cc:1136] tablet 72feac2a993141168ac9399d9361ef17: '8315d22c56f7439a9e1fc82f153f018c' -> '13d05ebdca8c4f489c597ce550dd773d' move completed: OK
I20240501 07:59:11.425371 9242 rebalancer_tool.cc:1136] tablet 3745acee2b5543b697a3c8f45d2618bb: '8b8b3ad3aa8f4053b2167dd550f2086f' -> 'a505e7cb133c4392bfda8a4a20874ae9' move completed: OK
I20240501 07:59:11.431630 9242 rebalancer_tool.cc:1136] tablet ff9dc384dbac47dc9795617c64688943: '3b164cbbc00f4e8eb3254ddd349a3ff9' -> '13d05ebdca8c4f489c597ce550dd773d' move completed: OK
I20240501 07:59:11.438452 9242 rebalancer_tool.cc:1123] tablet bab34f083c8045e4a6df373db5ebd54b: '8315d22c56f7439a9e1fc82f153f018c' -> '13d05ebdca8c4f489c597ce550dd773d' move is abandoned: Not found: bab34f083c8045e4a6df373db5ebd54b: tablet not found
I20240501 07:59:11.439260 9242 rebalancer_tool.cc:1084] tablet b914638d3a884ae4932316bdbd1fb4a4: '8b8b3ad3aa8f4053b2167dd550f2086f' -> '13d05ebdca8c4f489c597ce550dd773d' move ignored: Not found: b914638d3a884ae4932316bdbd1fb4a4: tablet not found
I20240501 07:59:11.439332 9242 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 07:59:11.717367 9242 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table d0e1f982440849beb494081bea9cb211: could not find any suitable replica to move from server 1f9cb72be98941b5a8e504cbe34132b9 to server 83f20e84af4c40d09a4804b6b26e697f
W20240501 07:59:11.717468 9242 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 4de37f887eb14e2dbe5d1609df1e538f: could not find any suitable replica to move from server 3b164cbbc00f4e8eb3254ddd349a3ff9 to server 74077b61d6f54342bf9736b4a9f2eade
W20240501 07:59:11.717504 9242 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 336b9f044a794b6a8a51d76e720b0ecd: could not find any suitable replica to move from server 3b164cbbc00f4e8eb3254ddd349a3ff9 to server 74077b61d6f54342bf9736b4a9f2eade
W20240501 07:59:11.717559 9242 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 5e1db4e06c5e470a9d2f8a20dd23c585: could not find any suitable replica to move from server 8315d22c56f7439a9e1fc82f153f018c to server 8b8b3ad3aa8f4053b2167dd550f2086f
W20240501 07:59:11.717588 9242 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 25ecaaa6eafe48ecb3f3ff353a6d1170: could not find any suitable replica to move from server 8b8b3ad3aa8f4053b2167dd550f2086f to server 74077b61d6f54342bf9736b4a9f2eade
W20240501 07:59:11.717639 9242 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 8e242b97203b46d0840cb3b6a649e5a7: could not find any suitable replica to move from server 3b164cbbc00f4e8eb3254ddd349a3ff9 to server 83f20e84af4c40d09a4804b6b26e697f
W20240501 07:59:11.717705 9242 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table bdae856e60144776aae709dddb837795: could not find any suitable replica to move from server 8b8b3ad3aa8f4053b2167dd550f2086f to server 13d05ebdca8c4f489c597ce550dd773d
W20240501 07:59:11.717736 9242 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3e062fd8d58941efbd3d190c28bb0059: could not find any suitable replica to move from server 8b8b3ad3aa8f4053b2167dd550f2086f to server 83f20e84af4c40d09a4804b6b26e697f
W20240501 07:59:11.717764 9242 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 8b10e9a527644d40b1515561fc5df5c2: could not find any suitable replica to move from server 74077b61d6f54342bf9736b4a9f2eade to server 8315d22c56f7439a9e1fc82f153f018c
W20240501 07:59:11.717815 9242 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 336b9f044a794b6a8a51d76e720b0ecd: could not find any suitable replica to move from server 8315d22c56f7439a9e1fc82f153f018c to server 74077b61d6f54342bf9736b4a9f2eade
I20240501 07:59:11.734958 9242 rebalancer_tool.cc:1074] tablet 7dd6566ccb6e43388146a6dfd1c4a88c: '3b164cbbc00f4e8eb3254ddd349a3ff9' -> '13d05ebdca8c4f489c597ce550dd773d' move scheduled
I20240501 07:59:11.752766 9242 rebalancer_tool.cc:1074] tablet 8454f24b0c28451b8ee51be39ca3e07f: '8b8b3ad3aa8f4053b2167dd550f2086f' -> '13d05ebdca8c4f489c597ce550dd773d' move scheduled
I20240501 07:59:11.768707 9242 rebalancer_tool.cc:1074] tablet ae46990350284295bf14aa64d4b6c93e: '8315d22c56f7439a9e1fc82f153f018c' -> 'a505e7cb133c4392bfda8a4a20874ae9' move scheduled
I20240501 07:59:11.793407 9242 rebalancer_tool.cc:1074] tablet 72aa4dc7664b48b7a73d34feba7d44d3: '74077b61d6f54342bf9736b4a9f2eade' -> '8315d22c56f7439a9e1fc82f153f018c' move scheduled
I20240501 07:59:11.877375 9242 rebalancer_tool.cc:1074] tablet 2976d6353e8947b7b6c37f72efbeeac4: '8b8b3ad3aa8f4053b2167dd550f2086f' -> '3b164cbbc00f4e8eb3254ddd349a3ff9' move scheduled
I20240501 07:59:11.997314 9242 rebalancer_tool.cc:1136] tablet 114b6dbafa1c4c44ac42f1d2293e3125: '1f9cb72be98941b5a8e504cbe34132b9' -> 'a505e7cb133c4392bfda8a4a20874ae9' move completed: OK
I20240501 07:59:12.026566 9242 rebalancer_tool.cc:1136] tablet b284ed2cb0264ced99a4ffd90983d902: '8b8b3ad3aa8f4053b2167dd550f2086f' -> '13d05ebdca8c4f489c597ce550dd773d' move completed: OK
I20240501 07:59:12.060582 9242 rebalancer_tool.cc:1136] tablet 6292ff0c7e624dd7b24715b83a30fff8: '83f20e84af4c40d09a4804b6b26e697f' -> '1f9cb72be98941b5a8e504cbe34132b9' move completed: OK
I20240501 07:59:12.072492 9242 rebalancer_tool.cc:1123] tablet 33b3834db0cd4407ae7c26e044459e87: '8315d22c56f7439a9e1fc82f153f018c' -> 'a505e7cb133c4392bfda8a4a20874ae9' move is abandoned: Not found: 33b3834db0cd4407ae7c26e044459e87: tablet not found
I20240501 07:59:12.098070 9242 rebalancer_tool.cc:1136] tablet 2398430a23094711a13a86cc88e27a78: '3b164cbbc00f4e8eb3254ddd349a3ff9' -> '74077b61d6f54342bf9736b4a9f2eade' move completed: OK
I20240501 07:59:12.120936 9242 rebalancer_tool.cc:1136] tablet e92d6aeeb3df4ea3852c6ebbed0e07c5: '3b164cbbc00f4e8eb3254ddd349a3ff9' -> '13d05ebdca8c4f489c597ce550dd773d' move completed: OK
I20240501 07:59:12.127429 9242 rebalancer_tool.cc:1136] tablet c3a9036ccfee45498bca9681518d3699: '1f9cb72be98941b5a8e504cbe34132b9' -> '74077b61d6f54342bf9736b4a9f2eade' move completed: OK
I20240501 07:59:12.140813 9242 rebalancer_tool.cc:1074] tablet df00e37199e944589a0a5f3feeee308a: '3b164cbbc00f4e8eb3254ddd349a3ff9' -> '13d05ebdca8c4f489c597ce550dd773d' move scheduled
I20240501 07:59:12.159638 9242 rebalancer_tool.cc:1074] tablet d3f2250a62c6429e886753aaffcbe883: '1f9cb72be98941b5a8e504cbe34132b9' -> 'a505e7cb133c4392bfda8a4a20874ae9' move scheduled
I20240501 07:59:12.161597 9242 rebalancer_tool.cc:1084] tablet 3b04063d87be40f7ac1b46715d457983: '1f9cb72be98941b5a8e504cbe34132b9' -> '74077b61d6f54342bf9736b4a9f2eade' move ignored: Not found: 3b04063d87be40f7ac1b46715d457983: tablet not found
I20240501 07:59:12.161666 9242 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 07:59:12.479774 9242 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 336b9f044a794b6a8a51d76e720b0ecd: could not find any suitable replica to move from server 3b164cbbc00f4e8eb3254ddd349a3ff9 to server 74077b61d6f54342bf9736b4a9f2eade
W20240501 07:59:12.479969 9242 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 5e1db4e06c5e470a9d2f8a20dd23c585: could not find any suitable replica to move from server 8315d22c56f7439a9e1fc82f153f018c to server 8b8b3ad3aa8f4053b2167dd550f2086f
W20240501 07:59:12.480041 9242 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 8b10e9a527644d40b1515561fc5df5c2: could not find any suitable replica to move from server a505e7cb133c4392bfda8a4a20874ae9 to server 8315d22c56f7439a9e1fc82f153f018c
W20240501 07:59:12.480089 9242 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table c4f46595186f4427aa00d6752aa5dc9a: could not find any suitable replica to move from server 1f9cb72be98941b5a8e504cbe34132b9 to server 83f20e84af4c40d09a4804b6b26e697f
W20240501 07:59:12.480132 9242 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 103f9d5ca0104a7ab2f2f75dead2f70f: could not find any suitable replica to move from server 13d05ebdca8c4f489c597ce550dd773d to server 3b164cbbc00f4e8eb3254ddd349a3ff9
W20240501 07:59:12.480198 9242 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 7f1dc10bb484406aa39a54213d4854b9: could not find any suitable replica to move from server a505e7cb133c4392bfda8a4a20874ae9 to server 74077b61d6f54342bf9736b4a9f2eade
I20240501 07:59:12.503990 9242 rebalancer_tool.cc:1074] tablet 239ee596e62d47169eeb6670dc62d59c: '74077b61d6f54342bf9736b4a9f2eade' -> '1f9cb72be98941b5a8e504cbe34132b9' move scheduled
I20240501 07:59:12.537178 9242 rebalancer_tool.cc:1074] tablet 8663a511521644009fc360ad92c093a1: '8315d22c56f7439a9e1fc82f153f018c' -> '13d05ebdca8c4f489c597ce550dd773d' move scheduled
I20240501 07:59:12.560151 9242 rebalancer_tool.cc:1074] tablet a756205204d146ce82a2799e30582f57: '3b164cbbc00f4e8eb3254ddd349a3ff9' -> 'a505e7cb133c4392bfda8a4a20874ae9' move scheduled
I20240501 07:59:12.587533 9242 rebalancer_tool.cc:1074] tablet ff9dc384dbac47dc9795617c64688943: '1f9cb72be98941b5a8e504cbe34132b9' -> '83f20e84af4c40d09a4804b6b26e697f' move scheduled
I20240501 07:59:12.611764 9242 rebalancer_tool.cc:1074] tablet d6c28cd91f624673b8e428f471a1dcce: '8b8b3ad3aa8f4053b2167dd550f2086f' -> 'a505e7cb133c4392bfda8a4a20874ae9' move scheduled
I20240501 07:59:12.678366 9242 rebalancer_tool.cc:1123] tablet ae46990350284295bf14aa64d4b6c93e: '8315d22c56f7439a9e1fc82f153f018c' -> 'a505e7cb133c4392bfda8a4a20874ae9' move is abandoned: Not found: ae46990350284295bf14aa64d4b6c93e: tablet not found
I20240501 07:59:12.704913 9242 rebalancer_tool.cc:1123] tablet aa1b8716228b4fd9af95870b7a613fb2: '1f9cb72be98941b5a8e504cbe34132b9' -> '83f20e84af4c40d09a4804b6b26e697f' move is abandoned: Not found: aa1b8716228b4fd9af95870b7a613fb2: tablet not found
I20240501 07:59:12.710268 9242 rebalancer_tool.cc:1136] tablet f68dbc1f65eb4f1b91a62e2de82a7676: '8315d22c56f7439a9e1fc82f153f018c' -> '83f20e84af4c40d09a4804b6b26e697f' move completed: OK
I20240501 07:59:12.722103 9242 rebalancer_tool.cc:1136] tablet fb6b46072a8d46e18b53b9f8de702859: '74077b61d6f54342bf9736b4a9f2eade' -> '8b8b3ad3aa8f4053b2167dd550f2086f' move completed: OK
I20240501 07:59:12.729009 9242 rebalancer_tool.cc:1136] tablet 6c3c0d6fe41f4ae196ae29beb9b68125: '1f9cb72be98941b5a8e504cbe34132b9' -> 'a505e7cb133c4392bfda8a4a20874ae9' move completed: OK
I20240501 07:59:12.755957 9242 rebalancer_tool.cc:1136] tablet 2e540f389b2a4aef93a020f549d11b67: '8315d22c56f7439a9e1fc82f153f018c' -> '83f20e84af4c40d09a4804b6b26e697f' move completed: OK
I20240501 07:59:12.779261 9242 rebalancer_tool.cc:1074] tablet dc550f54e8024d47b1955905e7858a1e: '8315d22c56f7439a9e1fc82f153f018c' -> '74077b61d6f54342bf9736b4a9f2eade' move scheduled
I20240501 07:59:12.795338 9242 rebalancer_tool.cc:1074] tablet f88fe033bce14bf0b1b63a52a7ba137e: '83f20e84af4c40d09a4804b6b26e697f' -> '8315d22c56f7439a9e1fc82f153f018c' move scheduled
I20240501 07:59:12.796911 9242 rebalancer_tool.cc:1084] tablet b284ed2cb0264ced99a4ffd90983d902: '1f9cb72be98941b5a8e504cbe34132b9' -> '74077b61d6f54342bf9736b4a9f2eade' move ignored: Not found: b284ed2cb0264ced99a4ffd90983d902: tablet not found
I20240501 07:59:12.797040 9242 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 07:59:13.082553 9242 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 4de37f887eb14e2dbe5d1609df1e538f: could not find any suitable replica to move from server 3b164cbbc00f4e8eb3254ddd349a3ff9 to server a505e7cb133c4392bfda8a4a20874ae9
W20240501 07:59:13.082737 9242 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 731fe27ae8e34d92bd595d43c4bc8f3a: could not find any suitable replica to move from server 74077b61d6f54342bf9736b4a9f2eade to server 8b8b3ad3aa8f4053b2167dd550f2086f
W20240501 07:59:13.082823 9242 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 731fe27ae8e34d92bd595d43c4bc8f3a: could not find any suitable replica to move from server 13d05ebdca8c4f489c597ce550dd773d to server 3b164cbbc00f4e8eb3254ddd349a3ff9
I20240501 07:59:13.097404 9242 rebalancer_tool.cc:1084] tablet 32ecf214d6e14fa3be1b8eafd8c23fac: '13d05ebdca8c4f489c597ce550dd773d' -> '83f20e84af4c40d09a4804b6b26e697f' move ignored: Illegal state: Tablet not RUNNING: STOPPING
I20240501 07:59:13.097478 9242 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 07:59:13.461686 9242 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 4df593f62ca14b82ae95b3b691699a61: could not find any suitable replica to move from server a505e7cb133c4392bfda8a4a20874ae9 to server 3b164cbbc00f4e8eb3254ddd349a3ff9
W20240501 07:59:13.461812 9242 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 4de37f887eb14e2dbe5d1609df1e538f: could not find any suitable replica to move from server 3b164cbbc00f4e8eb3254ddd349a3ff9 to server a505e7cb133c4392bfda8a4a20874ae9
W20240501 07:59:13.462000 9242 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 731fe27ae8e34d92bd595d43c4bc8f3a: could not find any suitable replica to move from server 13d05ebdca8c4f489c597ce550dd773d to server 8315d22c56f7439a9e1fc82f153f018c
W20240501 07:59:13.462040 9242 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 4df593f62ca14b82ae95b3b691699a61: could not find any suitable replica to move from server a505e7cb133c4392bfda8a4a20874ae9 to server 8315d22c56f7439a9e1fc82f153f018c
W20240501 07:59:13.462078 9242 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 103f9d5ca0104a7ab2f2f75dead2f70f: could not find any suitable replica to move from server 13d05ebdca8c4f489c597ce550dd773d to server 1f9cb72be98941b5a8e504cbe34132b9
W20240501 07:59:13.462159 9242 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table e37d9a776ed7492ba6f616e4727f982b: could not find any suitable replica to move from server a505e7cb133c4392bfda8a4a20874ae9 to server 83f20e84af4c40d09a4804b6b26e697f
I20240501 07:59:13.482168 9242 rebalancer_tool.cc:1074] tablet 6dc929e2a6ae45bd8b1f297583636bb6: '8315d22c56f7439a9e1fc82f153f018c' -> '83f20e84af4c40d09a4804b6b26e697f' move scheduled
I20240501 07:59:13.495817 9242 rebalancer_tool.cc:1074] tablet 6daebcfc41754ec4aca939ca20588854: '13d05ebdca8c4f489c597ce550dd773d' -> '8b8b3ad3aa8f4053b2167dd550f2086f' move scheduled
I20240501 07:59:13.561378 9242 rebalancer_tool.cc:1136] tablet d3f2250a62c6429e886753aaffcbe883: '1f9cb72be98941b5a8e504cbe34132b9' -> 'a505e7cb133c4392bfda8a4a20874ae9' move completed: OK
I20240501 07:59:13.573724 9242 rebalancer_tool.cc:1136] tablet 2976d6353e8947b7b6c37f72efbeeac4: '8b8b3ad3aa8f4053b2167dd550f2086f' -> '3b164cbbc00f4e8eb3254ddd349a3ff9' move completed: OK
I20240501 07:59:13.579886 9242 rebalancer_tool.cc:1136] tablet 7dd6566ccb6e43388146a6dfd1c4a88c: '3b164cbbc00f4e8eb3254ddd349a3ff9' -> '13d05ebdca8c4f489c597ce550dd773d' move completed: OK
I20240501 07:59:13.594412 9242 rebalancer_tool.cc:1136] tablet 782e37a4ebb34a68b461a74115ae8703: '8b8b3ad3aa8f4053b2167dd550f2086f' -> '83f20e84af4c40d09a4804b6b26e697f' move completed: OK
I20240501 07:59:13.617403 9242 rebalancer_tool.cc:1136] tablet 8454f24b0c28451b8ee51be39ca3e07f: '8b8b3ad3aa8f4053b2167dd550f2086f' -> '13d05ebdca8c4f489c597ce550dd773d' move completed: OK
I20240501 07:59:13.644346 9242 rebalancer_tool.cc:1074] tablet 542c9ac08f7d4bec9b15a309c2e21364: 'a505e7cb133c4392bfda8a4a20874ae9' -> '3b164cbbc00f4e8eb3254ddd349a3ff9' move scheduled
I20240501 07:59:13.664003 9242 rebalancer_tool.cc:1074] tablet 54b3efabad7f4b608cc9d931546b5743: '83f20e84af4c40d09a4804b6b26e697f' -> '8b8b3ad3aa8f4053b2167dd550f2086f' move scheduled
I20240501 07:59:13.698545 9242 rebalancer_tool.cc:1074] tablet 2e540f389b2a4aef93a020f549d11b67: '3b164cbbc00f4e8eb3254ddd349a3ff9' -> '13d05ebdca8c4f489c597ce550dd773d' move scheduled
I20240501 07:59:14.164883 9242 rebalancer_tool.cc:1136] tablet a756205204d146ce82a2799e30582f57: '3b164cbbc00f4e8eb3254ddd349a3ff9' -> 'a505e7cb133c4392bfda8a4a20874ae9' move completed: OK
I20240501 07:59:14.171299 9242 rebalancer_tool.cc:1136] tablet 8663a511521644009fc360ad92c093a1: '8315d22c56f7439a9e1fc82f153f018c' -> '13d05ebdca8c4f489c597ce550dd773d' move completed: OK
I20240501 07:59:14.187953 9242 rebalancer_tool.cc:1136] tablet df00e37199e944589a0a5f3feeee308a: '3b164cbbc00f4e8eb3254ddd349a3ff9' -> '13d05ebdca8c4f489c597ce550dd773d' move completed: OK
I20240501 07:59:14.224565 9242 rebalancer_tool.cc:1136] tablet 239ee596e62d47169eeb6670dc62d59c: '74077b61d6f54342bf9736b4a9f2eade' -> '1f9cb72be98941b5a8e504cbe34132b9' move completed: OK
I20240501 07:59:14.248764 9242 rebalancer_tool.cc:1136] tablet 72aa4dc7664b48b7a73d34feba7d44d3: '74077b61d6f54342bf9736b4a9f2eade' -> '8315d22c56f7439a9e1fc82f153f018c' move completed: OK
I20240501 07:59:14.279583 9242 rebalancer_tool.cc:1074] tablet 6c3c0d6fe41f4ae196ae29beb9b68125: '3b164cbbc00f4e8eb3254ddd349a3ff9' -> '13d05ebdca8c4f489c597ce550dd773d' move scheduled
I20240501 07:59:14.298827 9242 rebalancer_tool.cc:1074] tablet 114b6dbafa1c4c44ac42f1d2293e3125: '8315d22c56f7439a9e1fc82f153f018c' -> '74077b61d6f54342bf9736b4a9f2eade' move scheduled
I20240501 07:59:14.321810 9242 rebalancer_tool.cc:1074] tablet 2f9e82c3f1df4d1292e37e9cea07186e: '13d05ebdca8c4f489c597ce550dd773d' -> '8315d22c56f7439a9e1fc82f153f018c' move scheduled
I20240501 07:59:14.353070 9242 rebalancer_tool.cc:1074] tablet 28eaf22342ff4b5394f958d0b659273c: '74077b61d6f54342bf9736b4a9f2eade' -> '3b164cbbc00f4e8eb3254ddd349a3ff9' move scheduled
I20240501 07:59:14.449232 9242 rebalancer_tool.cc:1136] tablet dc550f54e8024d47b1955905e7858a1e: '8315d22c56f7439a9e1fc82f153f018c' -> '74077b61d6f54342bf9736b4a9f2eade' move completed: OK
I20240501 07:59:14.455575 9242 rebalancer_tool.cc:1136] tablet d6c28cd91f624673b8e428f471a1dcce: '8b8b3ad3aa8f4053b2167dd550f2086f' -> 'a505e7cb133c4392bfda8a4a20874ae9' move completed: OK
I20240501 07:59:14.489313 9242 rebalancer_tool.cc:1136] tablet f88fe033bce14bf0b1b63a52a7ba137e: '83f20e84af4c40d09a4804b6b26e697f' -> '8315d22c56f7439a9e1fc82f153f018c' move completed: OK
I20240501 07:59:14.533910 9242 rebalancer_tool.cc:1074] tablet 6221deb431d94bfca1c14696e0e0a0c5: '8315d22c56f7439a9e1fc82f153f018c' -> '74077b61d6f54342bf9736b4a9f2eade' move scheduled
I20240501 07:59:14.556803 9242 rebalancer_tool.cc:1074] tablet a4727883b2544ed58b356f41dc8017a9: '8315d22c56f7439a9e1fc82f153f018c' -> '13d05ebdca8c4f489c597ce550dd773d' move scheduled
I20240501 07:59:14.675029 9242 rebalancer_tool.cc:1136] tablet 6e91683ff22243c1b3877733d74e2886: '3b164cbbc00f4e8eb3254ddd349a3ff9' -> '74077b61d6f54342bf9736b4a9f2eade' move completed: OK
I20240501 07:59:14.700778 9242 rebalancer_tool.cc:1074] tablet 59e171892fc1414fbcfd6a8c090854f3: '83f20e84af4c40d09a4804b6b26e697f' -> '3b164cbbc00f4e8eb3254ddd349a3ff9' move scheduled
I20240501 07:59:15.068147 9242 rebalancer_tool.cc:1136] tablet 2e540f389b2a4aef93a020f549d11b67: '3b164cbbc00f4e8eb3254ddd349a3ff9' -> '13d05ebdca8c4f489c597ce550dd773d' move completed: OK
I20240501 07:59:15.079941 9242 rebalancer_tool.cc:1136] tablet 54b3efabad7f4b608cc9d931546b5743: '83f20e84af4c40d09a4804b6b26e697f' -> '8b8b3ad3aa8f4053b2167dd550f2086f' move completed: OK
I20240501 07:59:15.125900 9242 rebalancer_tool.cc:1136] tablet 6daebcfc41754ec4aca939ca20588854: '13d05ebdca8c4f489c597ce550dd773d' -> '8b8b3ad3aa8f4053b2167dd550f2086f' move completed: OK
I20240501 07:59:15.161474 9242 rebalancer_tool.cc:1074] tablet 12ce6a21d06c4ba889542758be9dc62d: '74077b61d6f54342bf9736b4a9f2eade' -> '3b164cbbc00f4e8eb3254ddd349a3ff9' move scheduled
I20240501 07:59:15.306984 9242 rebalancer_tool.cc:1136] tablet 542c9ac08f7d4bec9b15a309c2e21364: 'a505e7cb133c4392bfda8a4a20874ae9' -> '3b164cbbc00f4e8eb3254ddd349a3ff9' move completed: OK
I20240501 07:59:15.334028 9242 rebalancer_tool.cc:1136] tablet ff9dc384dbac47dc9795617c64688943: '1f9cb72be98941b5a8e504cbe34132b9' -> '83f20e84af4c40d09a4804b6b26e697f' move completed: OK
I20240501 07:59:15.350085 9242 rebalancer_tool.cc:1074] tablet b7d57b7fd5234f50afbc859bc997f3e1: '3b164cbbc00f4e8eb3254ddd349a3ff9' -> '83f20e84af4c40d09a4804b6b26e697f' move scheduled
I20240501 07:59:15.980705 9242 rebalancer_tool.cc:1136] tablet a4727883b2544ed58b356f41dc8017a9: '8315d22c56f7439a9e1fc82f153f018c' -> '13d05ebdca8c4f489c597ce550dd773d' move completed: OK
I20240501 07:59:16.006129 9242 rebalancer_tool.cc:1136] tablet 28eaf22342ff4b5394f958d0b659273c: '74077b61d6f54342bf9736b4a9f2eade' -> '3b164cbbc00f4e8eb3254ddd349a3ff9' move completed: OK
I20240501 07:59:16.027630 9242 rebalancer_tool.cc:1136] tablet 6dc929e2a6ae45bd8b1f297583636bb6: '8315d22c56f7439a9e1fc82f153f018c' -> '83f20e84af4c40d09a4804b6b26e697f' move completed: OK
I20240501 07:59:16.034166 9242 rebalancer_tool.cc:1136] tablet 2f9e82c3f1df4d1292e37e9cea07186e: '13d05ebdca8c4f489c597ce550dd773d' -> '8315d22c56f7439a9e1fc82f153f018c' move completed: OK
I20240501 07:59:16.055508 9242 rebalancer_tool.cc:1074] tablet c3a9036ccfee45498bca9681518d3699: '3b164cbbc00f4e8eb3254ddd349a3ff9' -> '83f20e84af4c40d09a4804b6b26e697f' move scheduled
I20240501 07:59:16.144407 9242 rebalancer_tool.cc:1136] tablet 6c3c0d6fe41f4ae196ae29beb9b68125: '3b164cbbc00f4e8eb3254ddd349a3ff9' -> '13d05ebdca8c4f489c597ce550dd773d' move completed: OK
I20240501 07:59:16.165587 9242 rebalancer_tool.cc:1074] tablet c5a876dfb598428ab350593129505504: '8315d22c56f7439a9e1fc82f153f018c' -> '3b164cbbc00f4e8eb3254ddd349a3ff9' move scheduled
I20240501 07:59:16.443737 9242 rebalancer_tool.cc:1136] tablet 59e171892fc1414fbcfd6a8c090854f3: '83f20e84af4c40d09a4804b6b26e697f' -> '3b164cbbc00f4e8eb3254ddd349a3ff9' move completed: OK
I20240501 07:59:16.451452 9242 rebalancer_tool.cc:1136] tablet 114b6dbafa1c4c44ac42f1d2293e3125: '8315d22c56f7439a9e1fc82f153f018c' -> '74077b61d6f54342bf9736b4a9f2eade' move completed: OK
I20240501 07:59:16.476771 9242 rebalancer_tool.cc:1136] tablet 6221deb431d94bfca1c14696e0e0a0c5: '8315d22c56f7439a9e1fc82f153f018c' -> '74077b61d6f54342bf9736b4a9f2eade' move completed: OK
I20240501 07:59:16.491990 9242 rebalancer_tool.cc:1074] tablet 5ee738571af740c1b494d4f17413d1d8: '3b164cbbc00f4e8eb3254ddd349a3ff9' -> '83f20e84af4c40d09a4804b6b26e697f' move scheduled
I20240501 07:59:16.772454 9242 rebalancer_tool.cc:1136] tablet 12ce6a21d06c4ba889542758be9dc62d: '74077b61d6f54342bf9736b4a9f2eade' -> '3b164cbbc00f4e8eb3254ddd349a3ff9' move completed: OK
I20240501 07:59:17.508981 9242 rebalancer_tool.cc:1136] tablet c3a9036ccfee45498bca9681518d3699: '3b164cbbc00f4e8eb3254ddd349a3ff9' -> '83f20e84af4c40d09a4804b6b26e697f' move completed: OK
I20240501 07:59:17.973842 9242 rebalancer_tool.cc:1136] tablet c5a876dfb598428ab350593129505504: '8315d22c56f7439a9e1fc82f153f018c' -> '3b164cbbc00f4e8eb3254ddd349a3ff9' move completed: OK
I20240501 07:59:17.982074 9242 rebalancer_tool.cc:1136] tablet b7d57b7fd5234f50afbc859bc997f3e1: '3b164cbbc00f4e8eb3254ddd349a3ff9' -> '83f20e84af4c40d09a4804b6b26e697f' move completed: OK
I20240501 07:59:18.197794 9242 rebalancer_tool.cc:1136] tablet 5ee738571af740c1b494d4f17413d1d8: '3b164cbbc00f4e8eb3254ddd349a3ff9' -> '83f20e84af4c40d09a4804b6b26e697f' move completed: OK
I20240501 07:59:18.492991 9242 rebalancer_tool.cc:1074] tablet 4d4f47308d174a92bc73f23f2242be9a: 'a505e7cb133c4392bfda8a4a20874ae9' -> '8b8b3ad3aa8f4053b2167dd550f2086f' move scheduled
I20240501 07:59:18.510680 9242 rebalancer_tool.cc:1074] tablet 144ff11d2a504f7ab7a230e6822d24ca: '83f20e84af4c40d09a4804b6b26e697f' -> '8315d22c56f7439a9e1fc82f153f018c' move scheduled
I20240501 07:59:18.530206 9242 rebalancer_tool.cc:1074] tablet 36aaf192042f4320b2526322424dc640: '13d05ebdca8c4f489c597ce550dd773d' -> '1f9cb72be98941b5a8e504cbe34132b9' move scheduled
I20240501 07:59:18.545197 9242 rebalancer_tool.cc:1074] tablet 2560b1ec81ac4357b32c7a8ba9d72216: '74077b61d6f54342bf9736b4a9f2eade' -> '3b164cbbc00f4e8eb3254ddd349a3ff9' move scheduled
I20240501 07:59:18.563838 9242 rebalancer_tool.cc:1074] tablet a93ce5841c7f40a9ab23f88efe7b85bb: 'a505e7cb133c4392bfda8a4a20874ae9' -> '8b8b3ad3aa8f4053b2167dd550f2086f' move scheduled
I20240501 07:59:18.584332 9242 rebalancer_tool.cc:1074] tablet 0b24f67cf1a64debb23ee6618c820d33: '74077b61d6f54342bf9736b4a9f2eade' -> '83f20e84af4c40d09a4804b6b26e697f' move scheduled
I20240501 07:59:18.609588 9242 rebalancer_tool.cc:1074] tablet 0864c290fafa488abefa93d41c0a2efe: '13d05ebdca8c4f489c597ce550dd773d' -> '8315d22c56f7439a9e1fc82f153f018c' move scheduled
I20240501 07:59:18.627147 9242 rebalancer_tool.cc:1074] tablet d3f2250a62c6429e886753aaffcbe883: '3b164cbbc00f4e8eb3254ddd349a3ff9' -> '1f9cb72be98941b5a8e504cbe34132b9' move scheduled
I20240501 07:59:18.653568 9242 rebalancer_tool.cc:1074] tablet c86c420310ec42ffa5b24304fba74267: 'a505e7cb133c4392bfda8a4a20874ae9' -> '74077b61d6f54342bf9736b4a9f2eade' move scheduled
I20240501 07:59:18.670009 9242 rebalancer_tool.cc:1074] tablet c16b7962c18d4e02b850c7c7c678eae4: '13d05ebdca8c4f489c597ce550dd773d' -> '8b8b3ad3aa8f4053b2167dd550f2086f' move scheduled
I20240501 07:59:18.688053 9242 rebalancer_tool.cc:1074] tablet a29720541b7b4a79aaa810592ef7c4b1: '83f20e84af4c40d09a4804b6b26e697f' -> 'a505e7cb133c4392bfda8a4a20874ae9' move scheduled
I20240501 07:59:18.722214 9242 rebalancer_tool.cc:1074] tablet e0583fa43994452ca7553cbf2309eba1: 'a505e7cb133c4392bfda8a4a20874ae9' -> '8315d22c56f7439a9e1fc82f153f018c' move scheduled
I20240501 07:59:18.736704 9242 rebalancer_tool.cc:1074] tablet 8f36eefcda3f475d9571e92fdeb52eb4: '13d05ebdca8c4f489c597ce550dd773d' -> '3b164cbbc00f4e8eb3254ddd349a3ff9' move scheduled
I20240501 07:59:18.753087 9242 rebalancer_tool.cc:1074] tablet 1c32cd00bc344ae1b161d3f9b43b53d3: '1f9cb72be98941b5a8e504cbe34132b9' -> '74077b61d6f54342bf9736b4a9f2eade' move scheduled
I20240501 07:59:18.767820 9242 rebalancer_tool.cc:1074] tablet e5d564b9558946f585ca7a7a3b457a07: '8b8b3ad3aa8f4053b2167dd550f2086f' -> '1f9cb72be98941b5a8e504cbe34132b9' move scheduled
I20240501 07:59:18.787501 9242 rebalancer_tool.cc:1074] tablet 12ce6a21d06c4ba889542758be9dc62d: '13d05ebdca8c4f489c597ce550dd773d' -> '1f9cb72be98941b5a8e504cbe34132b9' move scheduled
I20240501 07:59:18.803966 9242 rebalancer_tool.cc:1074] tablet 0f9e2549fe294a93b3b25081a15b2028: '74077b61d6f54342bf9736b4a9f2eade' -> '8b8b3ad3aa8f4053b2167dd550f2086f' move scheduled
W20240501 07:59:19.645956 13923 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 07:59:19.349169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.99.136:46149 (local address 127.0.0.1:45250)
0501 07:59:19.349396 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0501 07:59:19.349413 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0501 07:59:19.349517 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 07:59:19.645220 (+295703us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":69,"threads_started":1}
I20240501 07:59:19.646896 9242 rebalancer_tool.cc:1123] tablet a93ce5841c7f40a9ab23f88efe7b85bb: 'a505e7cb133c4392bfda8a4a20874ae9' -> '8b8b3ad3aa8f4053b2167dd550f2086f' move is abandoned: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 07:59:19.650774 9242 rebalancer_tool.cc:1123] tablet c86c420310ec42ffa5b24304fba74267: 'a505e7cb133c4392bfda8a4a20874ae9' -> '74077b61d6f54342bf9736b4a9f2eade' move is abandoned: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111)
W20240501 07:59:19.711967 13983 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 07:59:19.653013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.99.135:42607 (local address 127.0.0.1:41452)
0501 07:59:19.653239 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0501 07:59:19.653253 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0501 07:59:19.653343 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 07:59:19.711836 (+ 58493us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":90,"threads_started":1}
I20240501 07:59:19.712441 9242 rebalancer_tool.cc:1123] tablet 2560b1ec81ac4357b32c7a8ba9d72216: '74077b61d6f54342bf9736b4a9f2eade' -> '3b164cbbc00f4e8eb3254ddd349a3ff9' move is abandoned: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 07:59:19.730115 9242 rebalancer_tool.cc:1123] tablet c16b7962c18d4e02b850c7c7c678eae4: '13d05ebdca8c4f489c597ce550dd773d' -> '8b8b3ad3aa8f4053b2167dd550f2086f' move is abandoned: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111)
I20240501 07:59:19.732673 9242 rebalancer_tool.cc:1123] tablet 4d4f47308d174a92bc73f23f2242be9a: 'a505e7cb133c4392bfda8a4a20874ae9' -> '8b8b3ad3aa8f4053b2167dd550f2086f' move is abandoned: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111)
I20240501 07:59:19.732767 9242 rebalancer_tool.cc:756] re-synchronizing cluster state
Illegal state: tablet server 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): unacceptable health status UNAVAILABLE
W20240501 07:59:19.984102 8859 consensus_peers.cc:485] T ad3e528eaff349a4b3b03037652b1505 P 13d05ebdca8c4f489c597ce550dd773d -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:20.023921 13986 raft_consensus.cc:1050] T 1c32cd00bc344ae1b161d3f9b43b53d3 P 8b8b3ad3aa8f4053b2167dd550f2086f: attempting to promote NON_VOTER 74077b61d6f54342bf9736b4a9f2eade to VOTER
I20240501 07:59:20.024309 13986 consensus_queue.cc:237] T 1c32cd00bc344ae1b161d3f9b43b53d3 P 8b8b3ad3aa8f4053b2167dd550f2086f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { replace: true } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } }
I20240501 07:59:20.025611 6707 raft_consensus.cc:1261] T 1c32cd00bc344ae1b161d3f9b43b53d3 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Refusing update from remote peer 8b8b3ad3aa8f4053b2167dd550f2086f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 07:59:20.026219 13986 consensus_queue.cc:1035] T 1c32cd00bc344ae1b161d3f9b43b53d3 P 8b8b3ad3aa8f4053b2167dd550f2086f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 07:59:20.026659 8794 raft_consensus.cc:1261] T 1c32cd00bc344ae1b161d3f9b43b53d3 P 74077b61d6f54342bf9736b4a9f2eade [term 1 LEARNER]: Refusing update from remote peer 8b8b3ad3aa8f4053b2167dd550f2086f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
W20240501 07:59:20.026832 6249 consensus_peers.cc:485] T 1c32cd00bc344ae1b161d3f9b43b53d3 P 8b8b3ad3aa8f4053b2167dd550f2086f -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:20.027336 13424 consensus_queue.cc:1035] T 1c32cd00bc344ae1b161d3f9b43b53d3 P 8b8b3ad3aa8f4053b2167dd550f2086f [LEADER]: Connected to new peer: Peer: permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 07:59:20.031417 14068 raft_consensus.cc:2932] T 1c32cd00bc344ae1b161d3f9b43b53d3 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { replace: true } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } } }
I20240501 07:59:20.031908 6707 raft_consensus.cc:2932] T 1c32cd00bc344ae1b161d3f9b43b53d3 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { replace: true } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } } }
I20240501 07:59:20.033743 8794 raft_consensus.cc:2932] T 1c32cd00bc344ae1b161d3f9b43b53d3 P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { replace: true } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } } }
I20240501 07:59:20.036573 6169 catalog_manager.cc:5484] T 1c32cd00bc344ae1b161d3f9b43b53d3 P 1f9cb72be98941b5a8e504cbe34132b9 reported cstate change: config changed from index 2 to 3, 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { replace: true } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } } }
I20240501 07:59:20.075207 12119 raft_consensus.cc:1050] T d3f2250a62c6429e886753aaffcbe883 P 3b164cbbc00f4e8eb3254ddd349a3ff9: attempting to promote NON_VOTER 1f9cb72be98941b5a8e504cbe34132b9 to VOTER
I20240501 07:59:20.075686 12119 consensus_queue.cc:237] T d3f2250a62c6429e886753aaffcbe883 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { replace: true } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { promote: false } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { promote: false } }
I20240501 07:59:20.077091 6707 raft_consensus.cc:1261] T d3f2250a62c6429e886753aaffcbe883 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 LEARNER]: Refusing update from remote peer 3b164cbbc00f4e8eb3254ddd349a3ff9: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
W20240501 07:59:20.078100 6511 consensus_peers.cc:485] T d3f2250a62c6429e886753aaffcbe883 P 3b164cbbc00f4e8eb3254ddd349a3ff9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 07:59:20.078187 6511 consensus_peers.cc:485] T d3f2250a62c6429e886753aaffcbe883 P 3b164cbbc00f4e8eb3254ddd349a3ff9 -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:20.078536 14074 consensus_queue.cc:1035] T d3f2250a62c6429e886753aaffcbe883 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 07:59:20.079157 14074 raft_consensus.cc:1050] T 144ff11d2a504f7ab7a230e6822d24ca P 3b164cbbc00f4e8eb3254ddd349a3ff9: attempting to promote NON_VOTER 8315d22c56f7439a9e1fc82f153f018c to VOTER
I20240501 07:59:20.079552 14074 consensus_queue.cc:237] T 144ff11d2a504f7ab7a230e6822d24ca P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { replace: true } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { promote: false } }
I20240501 07:59:20.080910 6311 raft_consensus.cc:1261] T 144ff11d2a504f7ab7a230e6822d24ca P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Refusing update from remote peer 3b164cbbc00f4e8eb3254ddd349a3ff9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 07:59:20.081297 6444 raft_consensus.cc:1261] T 144ff11d2a504f7ab7a230e6822d24ca P 8315d22c56f7439a9e1fc82f153f018c [term 1 LEARNER]: Refusing update from remote peer 3b164cbbc00f4e8eb3254ddd349a3ff9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
W20240501 07:59:20.081624 6511 consensus_peers.cc:485] T 144ff11d2a504f7ab7a230e6822d24ca P 3b164cbbc00f4e8eb3254ddd349a3ff9 -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:20.081724 12119 consensus_queue.cc:1035] T 144ff11d2a504f7ab7a230e6822d24ca P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 07:59:20.081907 11107 consensus_queue.cc:1035] T 144ff11d2a504f7ab7a230e6822d24ca P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 07:59:20.083909 12119 raft_consensus.cc:2932] T 144ff11d2a504f7ab7a230e6822d24ca P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { replace: true } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { promote: false } } }
I20240501 07:59:20.084334 6311 raft_consensus.cc:2932] T 144ff11d2a504f7ab7a230e6822d24ca P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { replace: true } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { promote: false } } }
I20240501 07:59:20.087273 6444 raft_consensus.cc:2932] T 144ff11d2a504f7ab7a230e6822d24ca P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { replace: true } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { promote: false } } }
I20240501 07:59:20.087337 6165 catalog_manager.cc:5484] T 144ff11d2a504f7ab7a230e6822d24ca P 3b164cbbc00f4e8eb3254ddd349a3ff9 reported cstate change: config changed from index 2 to 3, 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 07:59:20.090266 6575 consensus_queue.cc:237] T 144ff11d2a504f7ab7a230e6822d24ca P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { promote: false } }
I20240501 07:59:20.091387 6311 raft_consensus.cc:1261] T 144ff11d2a504f7ab7a230e6822d24ca P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Refusing update from remote peer 3b164cbbc00f4e8eb3254ddd349a3ff9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:59:20.091727 6445 raft_consensus.cc:1261] T 144ff11d2a504f7ab7a230e6822d24ca P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Refusing update from remote peer 3b164cbbc00f4e8eb3254ddd349a3ff9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 07:59:20.092113 13956 consensus_queue.cc:1035] T 144ff11d2a504f7ab7a230e6822d24ca P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:20.092406 14074 consensus_queue.cc:1035] T 144ff11d2a504f7ab7a230e6822d24ca P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:20.095162 12119 raft_consensus.cc:2932] T 144ff11d2a504f7ab7a230e6822d24ca P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { promote: false } } }
I20240501 07:59:20.095587 6311 raft_consensus.cc:2932] T 144ff11d2a504f7ab7a230e6822d24ca P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { promote: false } } }
I20240501 07:59:20.098982 6160 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 144ff11d2a504f7ab7a230e6822d24ca with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 07:59:20.098832 6444 raft_consensus.cc:2932] T 144ff11d2a504f7ab7a230e6822d24ca P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { promote: false } } }
I20240501 07:59:20.099189 6172 catalog_manager.cc:5484] T 144ff11d2a504f7ab7a230e6822d24ca P 3b164cbbc00f4e8eb3254ddd349a3ff9 reported cstate change: config changed from index 3 to 4, VOTER 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136) evicted. New cstate: current_term: 1 leader_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 07:59:20.104171 13040 raft_consensus.cc:1050] T e0583fa43994452ca7553cbf2309eba1 P 13d05ebdca8c4f489c597ce550dd773d: attempting to promote NON_VOTER 8315d22c56f7439a9e1fc82f153f018c to VOTER
I20240501 07:59:20.104535 13040 consensus_queue.cc:237] T e0583fa43994452ca7553cbf2309eba1 P 13d05ebdca8c4f489c597ce550dd773d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { replace: true } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { promote: false } }
I20240501 07:59:20.105933 6444 raft_consensus.cc:1261] T e0583fa43994452ca7553cbf2309eba1 P 8315d22c56f7439a9e1fc82f153f018c [term 1 LEARNER]: Refusing update from remote peer 13d05ebdca8c4f489c597ce550dd773d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 07:59:20.106539 14081 consensus_queue.cc:1035] T e0583fa43994452ca7553cbf2309eba1 P 13d05ebdca8c4f489c597ce550dd773d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20240501 07:59:20.106806 8859 consensus_peers.cc:485] T e0583fa43994452ca7553cbf2309eba1 P 13d05ebdca8c4f489c597ce550dd773d -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 07:59:20.106904 8859 consensus_peers.cc:485] T e0583fa43994452ca7553cbf2309eba1 P 13d05ebdca8c4f489c597ce550dd773d -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:20.112411 6169 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55950:
name: "rebalancer_extra_table_32"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 07:59:20.118474 6556 tablet_service.cc:1467] Processing CreateTablet for tablet f788de4df5454d00ac9bb883f0abc9bd (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=388a7a41853a4416a7bbc17735cdbe8a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:20.118813 6556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f788de4df5454d00ac9bb883f0abc9bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:20.119171 6295 tablet_service.cc:1467] Processing CreateTablet for tablet b4a2db0198774bb2b4f3df769260e32f (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=388a7a41853a4416a7bbc17735cdbe8a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:20.119432 6295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4a2db0198774bb2b4f3df769260e32f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:20.121344 14084 tablet_bootstrap.cc:492] T f788de4df5454d00ac9bb883f0abc9bd P 3b164cbbc00f4e8eb3254ddd349a3ff9: Bootstrap starting.
I20240501 07:59:20.121551 8905 tablet_service.cc:1467] Processing CreateTablet for tablet f788de4df5454d00ac9bb883f0abc9bd (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=388a7a41853a4416a7bbc17735cdbe8a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:20.121798 14085 tablet_bootstrap.cc:492] T b4a2db0198774bb2b4f3df769260e32f P 8b8b3ad3aa8f4053b2167dd550f2086f: Bootstrap starting.
I20240501 07:59:20.121835 8905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f788de4df5454d00ac9bb883f0abc9bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:20.122709 14085 tablet_bootstrap.cc:654] T b4a2db0198774bb2b4f3df769260e32f P 8b8b3ad3aa8f4053b2167dd550f2086f: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:20.122829 14084 tablet_bootstrap.cc:654] T f788de4df5454d00ac9bb883f0abc9bd P 3b164cbbc00f4e8eb3254ddd349a3ff9: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:20.123838 14085 tablet_bootstrap.cc:492] T b4a2db0198774bb2b4f3df769260e32f P 8b8b3ad3aa8f4053b2167dd550f2086f: No bootstrap required, opened a new log
I20240501 07:59:20.123917 14085 ts_tablet_manager.cc:1367] T b4a2db0198774bb2b4f3df769260e32f P 8b8b3ad3aa8f4053b2167dd550f2086f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 07:59:20.123978 14088 tablet_bootstrap.cc:492] T f788de4df5454d00ac9bb883f0abc9bd P 13d05ebdca8c4f489c597ce550dd773d: Bootstrap starting.
I20240501 07:59:20.124312 14085 raft_consensus.cc:348] T b4a2db0198774bb2b4f3df769260e32f P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } }
I20240501 07:59:20.124419 14085 raft_consensus.cc:374] T b4a2db0198774bb2b4f3df769260e32f P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:20.124346 8757 tablet_service.cc:1467] Processing CreateTablet for tablet 8a39f168c8f846cd9b8c4e2f2cf62fdd (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=388a7a41853a4416a7bbc17735cdbe8a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:20.124451 14085 raft_consensus.cc:729] T b4a2db0198774bb2b4f3df769260e32f P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b8b3ad3aa8f4053b2167dd550f2086f, State: Initialized, Role: FOLLOWER
I20240501 07:59:20.124627 8757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a39f168c8f846cd9b8c4e2f2cf62fdd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:20.124990 14084 tablet_bootstrap.cc:492] T f788de4df5454d00ac9bb883f0abc9bd P 3b164cbbc00f4e8eb3254ddd349a3ff9: No bootstrap required, opened a new log
I20240501 07:59:20.124948 8904 tablet_service.cc:1467] Processing CreateTablet for tablet 8a39f168c8f846cd9b8c4e2f2cf62fdd (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=388a7a41853a4416a7bbc17735cdbe8a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:20.125082 14084 ts_tablet_manager.cc:1367] T f788de4df5454d00ac9bb883f0abc9bd P 3b164cbbc00f4e8eb3254ddd349a3ff9: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 07:59:20.125061 8905 tablet_service.cc:1467] Processing CreateTablet for tablet b4a2db0198774bb2b4f3df769260e32f (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=388a7a41853a4416a7bbc17735cdbe8a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:20.125217 8904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a39f168c8f846cd9b8c4e2f2cf62fdd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:20.125407 8905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4a2db0198774bb2b4f3df769260e32f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:20.125492 14084 raft_consensus.cc:348] T f788de4df5454d00ac9bb883f0abc9bd P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } }
I20240501 07:59:20.125604 14084 raft_consensus.cc:374] T f788de4df5454d00ac9bb883f0abc9bd P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:20.125633 14084 raft_consensus.cc:729] T f788de4df5454d00ac9bb883f0abc9bd P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b164cbbc00f4e8eb3254ddd349a3ff9, State: Initialized, Role: FOLLOWER
I20240501 07:59:20.125777 14084 consensus_queue.cc:260] T f788de4df5454d00ac9bb883f0abc9bd P 3b164cbbc00f4e8eb3254ddd349a3ff9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } }
I20240501 07:59:20.126073 14084 ts_tablet_manager.cc:1398] T f788de4df5454d00ac9bb883f0abc9bd P 3b164cbbc00f4e8eb3254ddd349a3ff9: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:59:20.126284 14085 consensus_queue.cc:260] T b4a2db0198774bb2b4f3df769260e32f P 8b8b3ad3aa8f4053b2167dd550f2086f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } }
I20240501 07:59:20.126605 14085 ts_tablet_manager.cc:1398] T b4a2db0198774bb2b4f3df769260e32f P 8b8b3ad3aa8f4053b2167dd550f2086f: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 07:59:20.127795 14088 tablet_bootstrap.cc:654] T f788de4df5454d00ac9bb883f0abc9bd P 13d05ebdca8c4f489c597ce550dd773d: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:20.129004 14088 tablet_bootstrap.cc:492] T f788de4df5454d00ac9bb883f0abc9bd P 13d05ebdca8c4f489c597ce550dd773d: No bootstrap required, opened a new log
I20240501 07:59:20.129096 14088 ts_tablet_manager.cc:1367] T f788de4df5454d00ac9bb883f0abc9bd P 13d05ebdca8c4f489c597ce550dd773d: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 07:59:20.129571 14088 raft_consensus.cc:348] T f788de4df5454d00ac9bb883f0abc9bd P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } }
I20240501 07:59:20.129668 14088 raft_consensus.cc:374] T f788de4df5454d00ac9bb883f0abc9bd P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:20.129698 14088 raft_consensus.cc:729] T f788de4df5454d00ac9bb883f0abc9bd P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13d05ebdca8c4f489c597ce550dd773d, State: Initialized, Role: FOLLOWER
I20240501 07:59:20.129846 14088 consensus_queue.cc:260] T f788de4df5454d00ac9bb883f0abc9bd P 13d05ebdca8c4f489c597ce550dd773d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } }
I20240501 07:59:20.130115 14088 ts_tablet_manager.cc:1398] T f788de4df5454d00ac9bb883f0abc9bd P 13d05ebdca8c4f489c597ce550dd773d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:20.130275 14088 tablet_bootstrap.cc:492] T 8a39f168c8f846cd9b8c4e2f2cf62fdd P 13d05ebdca8c4f489c597ce550dd773d: Bootstrap starting.
I20240501 07:59:20.131062 14088 tablet_bootstrap.cc:654] T 8a39f168c8f846cd9b8c4e2f2cf62fdd P 13d05ebdca8c4f489c597ce550dd773d: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:20.132208 14088 tablet_bootstrap.cc:492] T 8a39f168c8f846cd9b8c4e2f2cf62fdd P 13d05ebdca8c4f489c597ce550dd773d: No bootstrap required, opened a new log
I20240501 07:59:20.132287 14088 ts_tablet_manager.cc:1367] T 8a39f168c8f846cd9b8c4e2f2cf62fdd P 13d05ebdca8c4f489c597ce550dd773d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 07:59:20.132791 14088 raft_consensus.cc:348] T 8a39f168c8f846cd9b8c4e2f2cf62fdd P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } }
I20240501 07:59:20.132889 14088 raft_consensus.cc:374] T 8a39f168c8f846cd9b8c4e2f2cf62fdd P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:20.132941 14088 raft_consensus.cc:729] T 8a39f168c8f846cd9b8c4e2f2cf62fdd P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13d05ebdca8c4f489c597ce550dd773d, State: Initialized, Role: FOLLOWER
I20240501 07:59:20.133080 14088 consensus_queue.cc:260] T 8a39f168c8f846cd9b8c4e2f2cf62fdd P 13d05ebdca8c4f489c597ce550dd773d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } }
I20240501 07:59:20.133369 14088 ts_tablet_manager.cc:1398] T 8a39f168c8f846cd9b8c4e2f2cf62fdd P 13d05ebdca8c4f489c597ce550dd773d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:20.135841 14088 tablet_bootstrap.cc:492] T b4a2db0198774bb2b4f3df769260e32f P 13d05ebdca8c4f489c597ce550dd773d: Bootstrap starting.
I20240501 07:59:20.136045 14089 tablet_bootstrap.cc:492] T 8a39f168c8f846cd9b8c4e2f2cf62fdd P 74077b61d6f54342bf9736b4a9f2eade: Bootstrap starting.
I20240501 07:59:20.136807 14088 tablet_bootstrap.cc:654] T b4a2db0198774bb2b4f3df769260e32f P 13d05ebdca8c4f489c597ce550dd773d: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:20.136945 14089 tablet_bootstrap.cc:654] T 8a39f168c8f846cd9b8c4e2f2cf62fdd P 74077b61d6f54342bf9736b4a9f2eade: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:20.138238 14089 tablet_bootstrap.cc:492] T 8a39f168c8f846cd9b8c4e2f2cf62fdd P 74077b61d6f54342bf9736b4a9f2eade: No bootstrap required, opened a new log
I20240501 07:59:20.138334 14089 ts_tablet_manager.cc:1367] T 8a39f168c8f846cd9b8c4e2f2cf62fdd P 74077b61d6f54342bf9736b4a9f2eade: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240501 07:59:20.138897 14089 raft_consensus.cc:348] T 8a39f168c8f846cd9b8c4e2f2cf62fdd P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } }
I20240501 07:59:20.139006 14089 raft_consensus.cc:374] T 8a39f168c8f846cd9b8c4e2f2cf62fdd P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:20.139040 14089 raft_consensus.cc:729] T 8a39f168c8f846cd9b8c4e2f2cf62fdd P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74077b61d6f54342bf9736b4a9f2eade, State: Initialized, Role: FOLLOWER
I20240501 07:59:20.139178 14089 consensus_queue.cc:260] T 8a39f168c8f846cd9b8c4e2f2cf62fdd P 74077b61d6f54342bf9736b4a9f2eade [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } }
I20240501 07:59:20.139458 14089 ts_tablet_manager.cc:1398] T 8a39f168c8f846cd9b8c4e2f2cf62fdd P 74077b61d6f54342bf9736b4a9f2eade: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 07:59:20.140022 14088 tablet_bootstrap.cc:492] T b4a2db0198774bb2b4f3df769260e32f P 13d05ebdca8c4f489c597ce550dd773d: No bootstrap required, opened a new log
I20240501 07:59:20.140107 14088 ts_tablet_manager.cc:1367] T b4a2db0198774bb2b4f3df769260e32f P 13d05ebdca8c4f489c597ce550dd773d: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 07:59:20.140558 14088 raft_consensus.cc:348] T b4a2db0198774bb2b4f3df769260e32f P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } }
I20240501 07:59:20.140681 14088 raft_consensus.cc:374] T b4a2db0198774bb2b4f3df769260e32f P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:20.140723 14088 raft_consensus.cc:729] T b4a2db0198774bb2b4f3df769260e32f P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13d05ebdca8c4f489c597ce550dd773d, State: Initialized, Role: FOLLOWER
I20240501 07:59:20.140847 14088 consensus_queue.cc:260] T b4a2db0198774bb2b4f3df769260e32f P 13d05ebdca8c4f489c597ce550dd773d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } }
I20240501 07:59:20.141165 14088 ts_tablet_manager.cc:1398] T b4a2db0198774bb2b4f3df769260e32f P 13d05ebdca8c4f489c597ce550dd773d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20240501 07:59:20.153391 6511 consensus_peers.cc:462] T 0b24f67cf1a64debb23ee6618c820d33 P 3b164cbbc00f4e8eb3254ddd349a3ff9 -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111)
I20240501 07:59:20.189275 13959 raft_consensus.cc:1050] T e5d564b9558946f585ca7a7a3b457a07 P 8b8b3ad3aa8f4053b2167dd550f2086f: attempting to promote NON_VOTER 1f9cb72be98941b5a8e504cbe34132b9 to VOTER
I20240501 07:59:20.189911 13959 consensus_queue.cc:237] T e5d564b9558946f585ca7a7a3b457a07 P 8b8b3ad3aa8f4053b2167dd550f2086f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { replace: true } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { promote: false } }
I20240501 07:59:20.191071 13396 raft_consensus.cc:482] T b7d57b7fd5234f50afbc859bc997f3e1 P 74077b61d6f54342bf9736b4a9f2eade [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 83f20e84af4c40d09a4804b6b26e697f)
I20240501 07:59:20.191188 13396 raft_consensus.cc:504] T b7d57b7fd5234f50afbc859bc997f3e1 P 74077b61d6f54342bf9736b4a9f2eade [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false } }
I20240501 07:59:20.191362 8794 raft_consensus.cc:1261] T e5d564b9558946f585ca7a7a3b457a07 P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Refusing update from remote peer 8b8b3ad3aa8f4053b2167dd550f2086f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 07:59:20.191413 8924 raft_consensus.cc:1261] T e5d564b9558946f585ca7a7a3b457a07 P 13d05ebdca8c4f489c597ce550dd773d [term 1 FOLLOWER]: Refusing update from remote peer 8b8b3ad3aa8f4053b2167dd550f2086f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 07:59:20.191556 13396 leader_election.cc:288] T b7d57b7fd5234f50afbc859bc997f3e1 P 74077b61d6f54342bf9736b4a9f2eade [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129:46875), 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149)
I20240501 07:59:20.191859 13424 consensus_queue.cc:1035] T e5d564b9558946f585ca7a7a3b457a07 P 8b8b3ad3aa8f4053b2167dd550f2086f [LEADER]: Connected to new peer: Peer: permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 07:59:20.191891 6311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7d57b7fd5234f50afbc859bc997f3e1" candidate_uuid: "74077b61d6f54342bf9736b4a9f2eade" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: false dest_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" is_pre_election: true
I20240501 07:59:20.192008 13986 consensus_queue.cc:1035] T e5d564b9558946f585ca7a7a3b457a07 P 8b8b3ad3aa8f4053b2167dd550f2086f [LEADER]: Connected to new peer: Peer: permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 07:59:20.192750 6707 raft_consensus.cc:1261] T e5d564b9558946f585ca7a7a3b457a07 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 LEARNER]: Refusing update from remote peer 8b8b3ad3aa8f4053b2167dd550f2086f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 07:59:20.193176 14081 raft_consensus.cc:482] T 8a39f168c8f846cd9b8c4e2f2cf62fdd P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:20.193264 14081 raft_consensus.cc:504] T 8a39f168c8f846cd9b8c4e2f2cf62fdd P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } }
I20240501 07:59:20.193657 14081 leader_election.cc:288] T 8a39f168c8f846cd9b8c4e2f2cf62fdd P 13d05ebdca8c4f489c597ce550dd773d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149), 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133:39859)
W20240501 07:59:20.193961 8728 leader_election.cc:334] T b7d57b7fd5234f50afbc859bc997f3e1 P 74077b61d6f54342bf9736b4a9f2eade [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111)
I20240501 07:59:20.194025 8728 leader_election.cc:302] T b7d57b7fd5234f50afbc859bc997f3e1 P 74077b61d6f54342bf9736b4a9f2eade [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74077b61d6f54342bf9736b4a9f2eade; no voters: 83f20e84af4c40d09a4804b6b26e697f, 8b8b3ad3aa8f4053b2167dd550f2086f
I20240501 07:59:20.194159 13396 raft_consensus.cc:2726] T b7d57b7fd5234f50afbc859bc997f3e1 P 74077b61d6f54342bf9736b4a9f2eade [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 07:59:20.194139 13986 raft_consensus.cc:2932] T e5d564b9558946f585ca7a7a3b457a07 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { replace: true } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { promote: false } } }
W20240501 07:59:20.194658 8859 leader_election.cc:334] T 8a39f168c8f846cd9b8c4e2f2cf62fdd P 13d05ebdca8c4f489c597ce550dd773d [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111)
I20240501 07:59:20.195204 8793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a39f168c8f846cd9b8c4e2f2cf62fdd" candidate_uuid: "13d05ebdca8c4f489c597ce550dd773d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74077b61d6f54342bf9736b4a9f2eade" is_pre_election: true
I20240501 07:59:20.195334 8793 raft_consensus.cc:2454] T 8a39f168c8f846cd9b8c4e2f2cf62fdd P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 13d05ebdca8c4f489c597ce550dd773d in term 0.
I20240501 07:59:20.195572 8858 leader_election.cc:302] T 8a39f168c8f846cd9b8c4e2f2cf62fdd P 13d05ebdca8c4f489c597ce550dd773d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 13d05ebdca8c4f489c597ce550dd773d, 74077b61d6f54342bf9736b4a9f2eade; no voters: 83f20e84af4c40d09a4804b6b26e697f
I20240501 07:59:20.195796 14081 raft_consensus.cc:2781] T 8a39f168c8f846cd9b8c4e2f2cf62fdd P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:20.195834 14081 raft_consensus.cc:482] T 8a39f168c8f846cd9b8c4e2f2cf62fdd P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:20.195874 14081 raft_consensus.cc:3037] T 8a39f168c8f846cd9b8c4e2f2cf62fdd P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:20.196133 8793 raft_consensus.cc:2932] T e5d564b9558946f585ca7a7a3b457a07 P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { replace: true } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { promote: false } } }
I20240501 07:59:20.196794 14081 raft_consensus.cc:504] T 8a39f168c8f846cd9b8c4e2f2cf62fdd P 13d05ebdca8c4f489c597ce550dd773d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } }
I20240501 07:59:20.197155 14081 leader_election.cc:288] T 8a39f168c8f846cd9b8c4e2f2cf62fdd P 13d05ebdca8c4f489c597ce550dd773d [CANDIDATE]: Term 1 election: Requested vote from peers 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149), 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133:39859)
I20240501 07:59:20.198030 13418 consensus_queue.cc:1035] T ad3e528eaff349a4b3b03037652b1505 P 13d05ebdca8c4f489c597ce550dd773d [LEADER]: Connected to new peer: Peer: permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:20.198483 14068 consensus_queue.cc:1035] T e5d564b9558946f585ca7a7a3b457a07 P 8b8b3ad3aa8f4053b2167dd550f2086f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 07:59:20.200057 6169 catalog_manager.cc:5484] T e5d564b9558946f585ca7a7a3b457a07 P 8b8b3ad3aa8f4053b2167dd550f2086f reported cstate change: config changed from index 2 to 3, 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 07:59:20.200630 6707 raft_consensus.cc:2932] T e5d564b9558946f585ca7a7a3b457a07 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { replace: true } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { promote: false } } }
I20240501 07:59:20.200675 8924 raft_consensus.cc:2932] T e5d564b9558946f585ca7a7a3b457a07 P 13d05ebdca8c4f489c597ce550dd773d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { replace: true } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { promote: false } } }
I20240501 07:59:20.202960 8793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a39f168c8f846cd9b8c4e2f2cf62fdd" candidate_uuid: "13d05ebdca8c4f489c597ce550dd773d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74077b61d6f54342bf9736b4a9f2eade"
I20240501 07:59:20.203080 8793 raft_consensus.cc:3037] T 8a39f168c8f846cd9b8c4e2f2cf62fdd P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Advancing to term 1
W20240501 07:59:20.203145 8859 leader_election.cc:334] T 8a39f168c8f846cd9b8c4e2f2cf62fdd P 13d05ebdca8c4f489c597ce550dd773d [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111)
I20240501 07:59:20.203994 8793 raft_consensus.cc:2454] T 8a39f168c8f846cd9b8c4e2f2cf62fdd P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13d05ebdca8c4f489c597ce550dd773d in term 1.
I20240501 07:59:20.204896 8858 leader_election.cc:302] T 8a39f168c8f846cd9b8c4e2f2cf62fdd P 13d05ebdca8c4f489c597ce550dd773d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 13d05ebdca8c4f489c597ce550dd773d, 74077b61d6f54342bf9736b4a9f2eade; no voters: 83f20e84af4c40d09a4804b6b26e697f
I20240501 07:59:20.205075 14081 raft_consensus.cc:482] T b4a2db0198774bb2b4f3df769260e32f P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:20.205158 14081 raft_consensus.cc:504] T b4a2db0198774bb2b4f3df769260e32f P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } }
I20240501 07:59:20.205260 13418 raft_consensus.cc:2781] T 8a39f168c8f846cd9b8c4e2f2cf62fdd P 13d05ebdca8c4f489c597ce550dd773d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:20.205354 13418 raft_consensus.cc:686] T 8a39f168c8f846cd9b8c4e2f2cf62fdd P 13d05ebdca8c4f489c597ce550dd773d [term 1 LEADER]: Becoming Leader. State: Replica: 13d05ebdca8c4f489c597ce550dd773d, State: Running, Role: LEADER
I20240501 07:59:20.205505 14081 leader_election.cc:288] T b4a2db0198774bb2b4f3df769260e32f P 13d05ebdca8c4f489c597ce550dd773d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607), 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129:46875)
I20240501 07:59:20.205817 6311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4a2db0198774bb2b4f3df769260e32f" candidate_uuid: "13d05ebdca8c4f489c597ce550dd773d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" is_pre_election: true
I20240501 07:59:20.205965 6311 raft_consensus.cc:2454] T b4a2db0198774bb2b4f3df769260e32f P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 13d05ebdca8c4f489c597ce550dd773d in term 0.
I20240501 07:59:20.205494 13418 consensus_queue.cc:237] T 8a39f168c8f846cd9b8c4e2f2cf62fdd P 13d05ebdca8c4f489c597ce550dd773d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } }
W20240501 07:59:20.208864 8859 leader_election.cc:334] T b4a2db0198774bb2b4f3df769260e32f P 13d05ebdca8c4f489c597ce550dd773d [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111)
I20240501 07:59:20.209153 8858 leader_election.cc:302] T b4a2db0198774bb2b4f3df769260e32f P 13d05ebdca8c4f489c597ce550dd773d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 13d05ebdca8c4f489c597ce550dd773d, 8b8b3ad3aa8f4053b2167dd550f2086f; no voters: a505e7cb133c4392bfda8a4a20874ae9
I20240501 07:59:20.209498 13040 raft_consensus.cc:2781] T b4a2db0198774bb2b4f3df769260e32f P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:20.209558 13040 raft_consensus.cc:482] T b4a2db0198774bb2b4f3df769260e32f P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:20.209591 13040 raft_consensus.cc:3037] T b4a2db0198774bb2b4f3df769260e32f P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:20.210506 13040 raft_consensus.cc:504] T b4a2db0198774bb2b4f3df769260e32f P 13d05ebdca8c4f489c597ce550dd773d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } }
I20240501 07:59:20.210803 13040 leader_election.cc:288] T b4a2db0198774bb2b4f3df769260e32f P 13d05ebdca8c4f489c597ce550dd773d [CANDIDATE]: Term 1 election: Requested vote from peers a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607), 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129:46875)
I20240501 07:59:20.211233 6172 catalog_manager.cc:5484] T 8a39f168c8f846cd9b8c4e2f2cf62fdd P 13d05ebdca8c4f489c597ce550dd773d reported cstate change: term changed from 0 to 1, leader changed from <none> to 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134). New cstate: current_term: 1 leader_uuid: "13d05ebdca8c4f489c597ce550dd773d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } health_report { overall_health: HEALTHY } } }
W20240501 07:59:20.211884 8859 leader_election.cc:334] T b4a2db0198774bb2b4f3df769260e32f P 13d05ebdca8c4f489c597ce550dd773d [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111)
I20240501 07:59:20.214244 6311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4a2db0198774bb2b4f3df769260e32f" candidate_uuid: "13d05ebdca8c4f489c597ce550dd773d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f"
I20240501 07:59:20.214382 6311 raft_consensus.cc:3037] T b4a2db0198774bb2b4f3df769260e32f P 8b8b3ad3aa8f4053b2167dd550f2086f [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:20.215494 6311 raft_consensus.cc:2454] T b4a2db0198774bb2b4f3df769260e32f P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13d05ebdca8c4f489c597ce550dd773d in term 1.
I20240501 07:59:20.216761 8858 leader_election.cc:302] T b4a2db0198774bb2b4f3df769260e32f P 13d05ebdca8c4f489c597ce550dd773d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 13d05ebdca8c4f489c597ce550dd773d, 8b8b3ad3aa8f4053b2167dd550f2086f; no voters: a505e7cb133c4392bfda8a4a20874ae9
I20240501 07:59:20.217010 14081 raft_consensus.cc:2781] T b4a2db0198774bb2b4f3df769260e32f P 13d05ebdca8c4f489c597ce550dd773d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:20.217103 14081 raft_consensus.cc:686] T b4a2db0198774bb2b4f3df769260e32f P 13d05ebdca8c4f489c597ce550dd773d [term 1 LEADER]: Becoming Leader. State: Replica: 13d05ebdca8c4f489c597ce550dd773d, State: Running, Role: LEADER
I20240501 07:59:20.217218 14081 consensus_queue.cc:237] T b4a2db0198774bb2b4f3df769260e32f P 13d05ebdca8c4f489c597ce550dd773d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } }
I20240501 07:59:20.220448 13396 raft_consensus.cc:1050] T 0f9e2549fe294a93b3b25081a15b2028 P 74077b61d6f54342bf9736b4a9f2eade: attempting to promote NON_VOTER 8b8b3ad3aa8f4053b2167dd550f2086f to VOTER
I20240501 07:59:20.220335 6172 catalog_manager.cc:5484] T b4a2db0198774bb2b4f3df769260e32f P 13d05ebdca8c4f489c597ce550dd773d reported cstate change: term changed from 0 to 1, leader changed from <none> to 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134). New cstate: current_term: 1 leader_uuid: "13d05ebdca8c4f489c597ce550dd773d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } health_report { overall_health: HEALTHY } } }
I20240501 07:59:20.220849 13396 consensus_queue.cc:237] T 0f9e2549fe294a93b3b25081a15b2028 P 74077b61d6f54342bf9736b4a9f2eade [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { promote: false } }
I20240501 07:59:20.223156 6575 raft_consensus.cc:1261] T 0f9e2549fe294a93b3b25081a15b2028 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Refusing update from remote peer 74077b61d6f54342bf9736b4a9f2eade: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 07:59:20.223572 13218 consensus_queue.cc:1035] T 0f9e2549fe294a93b3b25081a15b2028 P 74077b61d6f54342bf9736b4a9f2eade [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20240501 07:59:20.224797 8728 consensus_peers.cc:485] T 0f9e2549fe294a93b3b25081a15b2028 P 74077b61d6f54342bf9736b4a9f2eade -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:20.224987 6311 raft_consensus.cc:1261] T 0f9e2549fe294a93b3b25081a15b2028 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 LEARNER]: Refusing update from remote peer 74077b61d6f54342bf9736b4a9f2eade: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 07:59:20.225489 13218 consensus_queue.cc:1035] T 0f9e2549fe294a93b3b25081a15b2028 P 74077b61d6f54342bf9736b4a9f2eade [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 07:59:20.227360 13218 raft_consensus.cc:2932] T 0f9e2549fe294a93b3b25081a15b2028 P 74077b61d6f54342bf9736b4a9f2eade [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { promote: false } } }
I20240501 07:59:20.227797 6311 raft_consensus.cc:2932] T 0f9e2549fe294a93b3b25081a15b2028 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { promote: false } } }
I20240501 07:59:20.229652 6575 raft_consensus.cc:2932] T 0f9e2549fe294a93b3b25081a15b2028 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { promote: false } } }
I20240501 07:59:20.232056 6165 catalog_manager.cc:5484] T 0f9e2549fe294a93b3b25081a15b2028 P 74077b61d6f54342bf9736b4a9f2eade reported cstate change: config changed from index 2 to 3, 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "74077b61d6f54342bf9736b4a9f2eade" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 07:59:20.243361 11107 raft_consensus.cc:482] T c5a876dfb598428ab350593129505504 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83f20e84af4c40d09a4804b6b26e697f)
I20240501 07:59:20.243513 11107 raft_consensus.cc:504] T c5a876dfb598428ab350593129505504 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: false } }
I20240501 07:59:20.243896 11107 leader_election.cc:288] T c5a876dfb598428ab350593129505504 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149), 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133:39859)
I20240501 07:59:20.244340 8794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5a876dfb598428ab350593129505504" candidate_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "74077b61d6f54342bf9736b4a9f2eade" is_pre_election: true
W20240501 07:59:20.244908 6511 leader_election.cc:334] T c5a876dfb598428ab350593129505504 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111)
I20240501 07:59:20.245018 6511 leader_election.cc:302] T c5a876dfb598428ab350593129505504 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b164cbbc00f4e8eb3254ddd349a3ff9; no voters: 74077b61d6f54342bf9736b4a9f2eade, 83f20e84af4c40d09a4804b6b26e697f
I20240501 07:59:20.245162 11107 raft_consensus.cc:2726] T c5a876dfb598428ab350593129505504 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 07:59:20.273370 13396 raft_consensus.cc:482] T c5a876dfb598428ab350593129505504 P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83f20e84af4c40d09a4804b6b26e697f)
I20240501 07:59:20.273510 13396 raft_consensus.cc:504] T c5a876dfb598428ab350593129505504 P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: false } }
I20240501 07:59:20.273938 13396 leader_election.cc:288] T c5a876dfb598428ab350593129505504 P 74077b61d6f54342bf9736b4a9f2eade [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149), 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131:34471)
I20240501 07:59:20.274428 6575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5a876dfb598428ab350593129505504" candidate_uuid: "74077b61d6f54342bf9736b4a9f2eade" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" is_pre_election: true
I20240501 07:59:20.274582 6575 raft_consensus.cc:2454] T c5a876dfb598428ab350593129505504 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74077b61d6f54342bf9736b4a9f2eade in term 1.
I20240501 07:59:20.274906 8730 leader_election.cc:302] T c5a876dfb598428ab350593129505504 P 74077b61d6f54342bf9736b4a9f2eade [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b164cbbc00f4e8eb3254ddd349a3ff9, 74077b61d6f54342bf9736b4a9f2eade; no voters:
W20240501 07:59:20.275075 8728 leader_election.cc:334] T c5a876dfb598428ab350593129505504 P 74077b61d6f54342bf9736b4a9f2eade [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111)
I20240501 07:59:20.275085 13396 raft_consensus.cc:2781] T c5a876dfb598428ab350593129505504 P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 07:59:20.275198 13396 raft_consensus.cc:482] T c5a876dfb598428ab350593129505504 P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Starting leader election (detected failure of leader 83f20e84af4c40d09a4804b6b26e697f)
I20240501 07:59:20.275247 13396 raft_consensus.cc:3037] T c5a876dfb598428ab350593129505504 P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:20.276265 13396 raft_consensus.cc:504] T c5a876dfb598428ab350593129505504 P 74077b61d6f54342bf9736b4a9f2eade [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: false } }
I20240501 07:59:20.276702 13396 leader_election.cc:288] T c5a876dfb598428ab350593129505504 P 74077b61d6f54342bf9736b4a9f2eade [CANDIDATE]: Term 2 election: Requested vote from peers 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149), 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131:34471)
I20240501 07:59:20.276888 6575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5a876dfb598428ab350593129505504" candidate_uuid: "74077b61d6f54342bf9736b4a9f2eade" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9"
I20240501 07:59:20.277021 6575 raft_consensus.cc:3037] T c5a876dfb598428ab350593129505504 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Advancing to term 2
W20240501 07:59:20.277491 8728 leader_election.cc:334] T c5a876dfb598428ab350593129505504 P 74077b61d6f54342bf9736b4a9f2eade [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111)
I20240501 07:59:20.277966 6575 raft_consensus.cc:2454] T c5a876dfb598428ab350593129505504 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74077b61d6f54342bf9736b4a9f2eade in term 2.
I20240501 07:59:20.278223 8730 leader_election.cc:302] T c5a876dfb598428ab350593129505504 P 74077b61d6f54342bf9736b4a9f2eade [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b164cbbc00f4e8eb3254ddd349a3ff9, 74077b61d6f54342bf9736b4a9f2eade; no voters: 83f20e84af4c40d09a4804b6b26e697f
I20240501 07:59:20.278404 13396 raft_consensus.cc:2781] T c5a876dfb598428ab350593129505504 P 74077b61d6f54342bf9736b4a9f2eade [term 2 FOLLOWER]: Leader election won for term 2
I20240501 07:59:20.278494 13396 raft_consensus.cc:686] T c5a876dfb598428ab350593129505504 P 74077b61d6f54342bf9736b4a9f2eade [term 2 LEADER]: Becoming Leader. State: Replica: 74077b61d6f54342bf9736b4a9f2eade, State: Running, Role: LEADER
I20240501 07:59:20.278664 13396 consensus_queue.cc:237] T c5a876dfb598428ab350593129505504 P 74077b61d6f54342bf9736b4a9f2eade [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: false } }
I20240501 07:59:20.281312 6172 catalog_manager.cc:5484] T c5a876dfb598428ab350593129505504 P 74077b61d6f54342bf9736b4a9f2eade reported cstate change: term changed from 1 to 2, leader changed from 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136) to 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133). New cstate: current_term: 2 leader_uuid: "74077b61d6f54342bf9736b4a9f2eade" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20240501 07:59:20.290596 6642 consensus_peers.cc:462] T a29720541b7b4a79aaa810592ef7c4b1 P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111)
I20240501 07:59:20.301242 14081 raft_consensus.cc:1050] T 0864c290fafa488abefa93d41c0a2efe P 13d05ebdca8c4f489c597ce550dd773d: attempting to promote NON_VOTER 8315d22c56f7439a9e1fc82f153f018c to VOTER
I20240501 07:59:20.301707 14081 consensus_queue.cc:237] T 0864c290fafa488abefa93d41c0a2efe P 13d05ebdca8c4f489c597ce550dd773d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { replace: true } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { promote: false } }
I20240501 07:59:20.303306 8794 raft_consensus.cc:1261] T 0864c290fafa488abefa93d41c0a2efe P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Refusing update from remote peer 13d05ebdca8c4f489c597ce550dd773d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 07:59:20.303306 6444 raft_consensus.cc:1261] T 0864c290fafa488abefa93d41c0a2efe P 8315d22c56f7439a9e1fc82f153f018c [term 1 LEARNER]: Refusing update from remote peer 13d05ebdca8c4f489c597ce550dd773d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 07:59:20.303694 13040 consensus_queue.cc:1035] T 0864c290fafa488abefa93d41c0a2efe P 13d05ebdca8c4f489c597ce550dd773d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 07:59:20.303817 14081 consensus_queue.cc:1035] T 0864c290fafa488abefa93d41c0a2efe P 13d05ebdca8c4f489c597ce550dd773d [LEADER]: Connected to new peer: Peer: permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20240501 07:59:20.303969 8859 consensus_peers.cc:485] T 0864c290fafa488abefa93d41c0a2efe P 13d05ebdca8c4f489c597ce550dd773d -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:20.305915 14081 raft_consensus.cc:2932] T 0864c290fafa488abefa93d41c0a2efe P 13d05ebdca8c4f489c597ce550dd773d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { replace: true } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { promote: false } } }
I20240501 07:59:20.306319 6444 raft_consensus.cc:2932] T 0864c290fafa488abefa93d41c0a2efe P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { replace: true } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { promote: false } } }
I20240501 07:59:20.307641 8794 raft_consensus.cc:2932] T 0864c290fafa488abefa93d41c0a2efe P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { replace: true } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { promote: false } } }
I20240501 07:59:20.309113 6172 catalog_manager.cc:5484] T 0864c290fafa488abefa93d41c0a2efe P 8315d22c56f7439a9e1fc82f153f018c reported cstate change: config changed from index 2 to 3, 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "13d05ebdca8c4f489c597ce550dd773d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { replace: true } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { promote: false } } }
I20240501 07:59:20.354279 13172 raft_consensus.cc:482] T 1de93edea7714491ae339db8cdfec090 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a505e7cb133c4392bfda8a4a20874ae9)
I20240501 07:59:20.354460 13172 raft_consensus.cc:504] T 1de93edea7714491ae339db8cdfec090 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } }
I20240501 07:59:20.354897 13172 leader_election.cc:288] T 1de93edea7714491ae339db8cdfec090 P 8315d22c56f7439a9e1fc82f153f018c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149), a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607)
W20240501 07:59:20.356155 6380 leader_election.cc:334] T 1de93edea7714491ae339db8cdfec090 P 8315d22c56f7439a9e1fc82f153f018c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111)
W20240501 07:59:20.356305 6380 leader_election.cc:334] T 1de93edea7714491ae339db8cdfec090 P 8315d22c56f7439a9e1fc82f153f018c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111)
I20240501 07:59:20.356357 6380 leader_election.cc:302] T 1de93edea7714491ae339db8cdfec090 P 8315d22c56f7439a9e1fc82f153f018c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8315d22c56f7439a9e1fc82f153f018c; no voters: 83f20e84af4c40d09a4804b6b26e697f, a505e7cb133c4392bfda8a4a20874ae9
I20240501 07:59:20.356513 13172 raft_consensus.cc:2726] T 1de93edea7714491ae339db8cdfec090 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 07:59:20.379333 13956 raft_consensus.cc:482] T 2976d6353e8947b7b6c37f72efbeeac4 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83f20e84af4c40d09a4804b6b26e697f)
I20240501 07:59:20.379494 13956 raft_consensus.cc:504] T 2976d6353e8947b7b6c37f72efbeeac4 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: false } }
I20240501 07:59:20.379874 13956 leader_election.cc:288] T 2976d6353e8947b7b6c37f72efbeeac4 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607), 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149)
W20240501 07:59:20.381098 6511 leader_election.cc:334] T 2976d6353e8947b7b6c37f72efbeeac4 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111)
W20240501 07:59:20.381219 6511 leader_election.cc:334] T 2976d6353e8947b7b6c37f72efbeeac4 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111)
I20240501 07:59:20.381273 6511 leader_election.cc:302] T 2976d6353e8947b7b6c37f72efbeeac4 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b164cbbc00f4e8eb3254ddd349a3ff9; no voters: 83f20e84af4c40d09a4804b6b26e697f, a505e7cb133c4392bfda8a4a20874ae9
I20240501 07:59:20.381403 13956 raft_consensus.cc:2726] T 2976d6353e8947b7b6c37f72efbeeac4 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 07:59:20.408367 13396 raft_consensus.cc:482] T 4d4f47308d174a92bc73f23f2242be9a P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83f20e84af4c40d09a4804b6b26e697f)
I20240501 07:59:20.408524 13396 raft_consensus.cc:504] T 4d4f47308d174a92bc73f23f2242be9a P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { replace: true } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { promote: true } }
I20240501 07:59:20.409011 13396 leader_election.cc:288] T 4d4f47308d174a92bc73f23f2242be9a P 74077b61d6f54342bf9736b4a9f2eade [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149), a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607)
W20240501 07:59:20.410156 8728 leader_election.cc:334] T 4d4f47308d174a92bc73f23f2242be9a P 74077b61d6f54342bf9736b4a9f2eade [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111)
W20240501 07:59:20.410264 8728 leader_election.cc:334] T 4d4f47308d174a92bc73f23f2242be9a P 74077b61d6f54342bf9736b4a9f2eade [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111)
I20240501 07:59:20.410315 8728 leader_election.cc:302] T 4d4f47308d174a92bc73f23f2242be9a P 74077b61d6f54342bf9736b4a9f2eade [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74077b61d6f54342bf9736b4a9f2eade; no voters: 83f20e84af4c40d09a4804b6b26e697f, a505e7cb133c4392bfda8a4a20874ae9
I20240501 07:59:20.410475 13396 raft_consensus.cc:2726] T 4d4f47308d174a92bc73f23f2242be9a P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 07:59:20.443929 13942 raft_consensus.cc:482] T 28eaf22342ff4b5394f958d0b659273c P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a505e7cb133c4392bfda8a4a20874ae9)
I20240501 07:59:20.444094 13942 raft_consensus.cc:504] T 28eaf22342ff4b5394f958d0b659273c P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: false } }
I20240501 07:59:20.444162 13956 raft_consensus.cc:482] T 28eaf22342ff4b5394f958d0b659273c P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a505e7cb133c4392bfda8a4a20874ae9)
I20240501 07:59:20.444269 13956 raft_consensus.cc:504] T 28eaf22342ff4b5394f958d0b659273c P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: false } }
I20240501 07:59:20.444540 13942 leader_election.cc:288] T 28eaf22342ff4b5394f958d0b659273c P 1f9cb72be98941b5a8e504cbe34132b9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607), 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131:34471)
I20240501 07:59:20.444667 13956 leader_election.cc:288] T 28eaf22342ff4b5394f958d0b659273c P 3b164cbbc00f4e8eb3254ddd349a3ff9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607), 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132:36773)
I20240501 07:59:20.445379 6575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28eaf22342ff4b5394f958d0b659273c" candidate_uuid: "1f9cb72be98941b5a8e504cbe34132b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" is_pre_election: true
I20240501 07:59:20.445523 6575 raft_consensus.cc:2454] T 28eaf22342ff4b5394f958d0b659273c P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f9cb72be98941b5a8e504cbe34132b9 in term 1.
I20240501 07:59:20.445744 6707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28eaf22342ff4b5394f958d0b659273c" candidate_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1f9cb72be98941b5a8e504cbe34132b9" is_pre_election: true
I20240501 07:59:20.445865 6707 raft_consensus.cc:2454] T 28eaf22342ff4b5394f958d0b659273c P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b164cbbc00f4e8eb3254ddd349a3ff9 in term 1.
W20240501 07:59:20.447013 6511 leader_election.cc:334] T 28eaf22342ff4b5394f958d0b659273c P 3b164cbbc00f4e8eb3254ddd349a3ff9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111)
I20240501 07:59:20.447276 6512 leader_election.cc:302] T 28eaf22342ff4b5394f958d0b659273c P 3b164cbbc00f4e8eb3254ddd349a3ff9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1f9cb72be98941b5a8e504cbe34132b9, 3b164cbbc00f4e8eb3254ddd349a3ff9; no voters: a505e7cb133c4392bfda8a4a20874ae9
I20240501 07:59:20.447463 13956 raft_consensus.cc:2781] T 28eaf22342ff4b5394f958d0b659273c P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 07:59:20.447551 13956 raft_consensus.cc:482] T 28eaf22342ff4b5394f958d0b659273c P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Starting leader election (detected failure of leader a505e7cb133c4392bfda8a4a20874ae9)
I20240501 07:59:20.447628 13956 raft_consensus.cc:3037] T 28eaf22342ff4b5394f958d0b659273c P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:20.449036 6644 leader_election.cc:302] T 28eaf22342ff4b5394f958d0b659273c P 1f9cb72be98941b5a8e504cbe34132b9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f9cb72be98941b5a8e504cbe34132b9, 3b164cbbc00f4e8eb3254ddd349a3ff9; no voters:
I20240501 07:59:20.449309 13956 raft_consensus.cc:504] T 28eaf22342ff4b5394f958d0b659273c P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: false } }
W20240501 07:59:20.449502 6642 leader_election.cc:334] T 28eaf22342ff4b5394f958d0b659273c P 1f9cb72be98941b5a8e504cbe34132b9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111)
I20240501 07:59:20.449538 13418 raft_consensus.cc:482] T 72aa4dc7664b48b7a73d34feba7d44d3 P 13d05ebdca8c4f489c597ce550dd773d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a505e7cb133c4392bfda8a4a20874ae9)
I20240501 07:59:20.449693 13956 leader_election.cc:288] T 28eaf22342ff4b5394f958d0b659273c P 3b164cbbc00f4e8eb3254ddd349a3ff9 [CANDIDATE]: Term 2 election: Requested vote from peers a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607), 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132:36773)
I20240501 07:59:20.449646 13418 raft_consensus.cc:504] T 72aa4dc7664b48b7a73d34feba7d44d3 P 13d05ebdca8c4f489c597ce550dd773d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { promote: false } }
I20240501 07:59:20.449950 13942 raft_consensus.cc:2781] T 28eaf22342ff4b5394f958d0b659273c P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 07:59:20.450008 13942 raft_consensus.cc:482] T 28eaf22342ff4b5394f958d0b659273c P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Starting leader election (detected failure of leader a505e7cb133c4392bfda8a4a20874ae9)
I20240501 07:59:20.450050 13942 raft_consensus.cc:3037] T 28eaf22342ff4b5394f958d0b659273c P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:20.450057 13418 leader_election.cc:288] T 72aa4dc7664b48b7a73d34feba7d44d3 P 13d05ebdca8c4f489c597ce550dd773d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607), 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130:32941)
I20240501 07:59:20.450279 6707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28eaf22342ff4b5394f958d0b659273c" candidate_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1f9cb72be98941b5a8e504cbe34132b9"
I20240501 07:59:20.450965 6444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72aa4dc7664b48b7a73d34feba7d44d3" candidate_uuid: "13d05ebdca8c4f489c597ce550dd773d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8315d22c56f7439a9e1fc82f153f018c" is_pre_election: true
W20240501 07:59:20.452450 6511 leader_election.cc:334] T 28eaf22342ff4b5394f958d0b659273c P 3b164cbbc00f4e8eb3254ddd349a3ff9 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111)
W20240501 07:59:20.452903 8859 leader_election.cc:334] T 72aa4dc7664b48b7a73d34feba7d44d3 P 13d05ebdca8c4f489c597ce550dd773d [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111)
I20240501 07:59:20.452991 8859 leader_election.cc:302] T 72aa4dc7664b48b7a73d34feba7d44d3 P 13d05ebdca8c4f489c597ce550dd773d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 13d05ebdca8c4f489c597ce550dd773d; no voters: 8315d22c56f7439a9e1fc82f153f018c, a505e7cb133c4392bfda8a4a20874ae9
I20240501 07:59:20.453155 13418 raft_consensus.cc:2726] T 72aa4dc7664b48b7a73d34feba7d44d3 P 13d05ebdca8c4f489c597ce550dd773d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 07:59:20.453521 13942 raft_consensus.cc:504] T 28eaf22342ff4b5394f958d0b659273c P 1f9cb72be98941b5a8e504cbe34132b9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: false } }
I20240501 07:59:20.453764 6707 raft_consensus.cc:2379] T 28eaf22342ff4b5394f958d0b659273c P 1f9cb72be98941b5a8e504cbe34132b9 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 3b164cbbc00f4e8eb3254ddd349a3ff9 in current term 2: Already voted for candidate 1f9cb72be98941b5a8e504cbe34132b9 in this term.
I20240501 07:59:20.454401 6512 leader_election.cc:302] T 28eaf22342ff4b5394f958d0b659273c P 3b164cbbc00f4e8eb3254ddd349a3ff9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b164cbbc00f4e8eb3254ddd349a3ff9; no voters: 1f9cb72be98941b5a8e504cbe34132b9, a505e7cb133c4392bfda8a4a20874ae9
I20240501 07:59:20.454658 13956 raft_consensus.cc:2726] T 28eaf22342ff4b5394f958d0b659273c P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240501 07:59:20.456517 13942 leader_election.cc:288] T 28eaf22342ff4b5394f958d0b659273c P 1f9cb72be98941b5a8e504cbe34132b9 [CANDIDATE]: Term 2 election: Requested vote from peers a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607), 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131:34471)
W20240501 07:59:20.456825 6642 leader_election.cc:334] T 28eaf22342ff4b5394f958d0b659273c P 1f9cb72be98941b5a8e504cbe34132b9 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111)
I20240501 07:59:20.457016 6575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28eaf22342ff4b5394f958d0b659273c" candidate_uuid: "1f9cb72be98941b5a8e504cbe34132b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9"
I20240501 07:59:20.457159 6575 raft_consensus.cc:2379] T 28eaf22342ff4b5394f958d0b659273c P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 1f9cb72be98941b5a8e504cbe34132b9 in current term 2: Already voted for candidate 3b164cbbc00f4e8eb3254ddd349a3ff9 in this term.
I20240501 07:59:20.457445 6644 leader_election.cc:302] T 28eaf22342ff4b5394f958d0b659273c P 1f9cb72be98941b5a8e504cbe34132b9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1f9cb72be98941b5a8e504cbe34132b9; no voters: 3b164cbbc00f4e8eb3254ddd349a3ff9, a505e7cb133c4392bfda8a4a20874ae9
I20240501 07:59:20.457669 13942 raft_consensus.cc:2726] T 28eaf22342ff4b5394f958d0b659273c P 1f9cb72be98941b5a8e504cbe34132b9 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240501 07:59:20.460880 13396 raft_consensus.cc:482] T 2560b1ec81ac4357b32c7a8ba9d72216 P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a505e7cb133c4392bfda8a4a20874ae9)
I20240501 07:59:20.461012 13396 raft_consensus.cc:504] T 2560b1ec81ac4357b32c7a8ba9d72216 P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: true } }
I20240501 07:59:20.461421 13396 leader_election.cc:288] T 2560b1ec81ac4357b32c7a8ba9d72216 P 74077b61d6f54342bf9736b4a9f2eade [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132:36773), a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607)
I20240501 07:59:20.461789 6707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2560b1ec81ac4357b32c7a8ba9d72216" candidate_uuid: "74077b61d6f54342bf9736b4a9f2eade" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "1f9cb72be98941b5a8e504cbe34132b9" is_pre_election: true
I20240501 07:59:20.462020 13396 raft_consensus.cc:482] T a93ce5841c7f40a9ab23f88efe7b85bb P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83f20e84af4c40d09a4804b6b26e697f)
I20240501 07:59:20.462105 13396 raft_consensus.cc:504] T a93ce5841c7f40a9ab23f88efe7b85bb P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { replace: true } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { promote: true } }
I20240501 07:59:20.462430 13396 leader_election.cc:288] T a93ce5841c7f40a9ab23f88efe7b85bb P 74077b61d6f54342bf9736b4a9f2eade [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149), a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607)
W20240501 07:59:20.462706 8728 leader_election.cc:334] T 2560b1ec81ac4357b32c7a8ba9d72216 P 74077b61d6f54342bf9736b4a9f2eade [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111)
I20240501 07:59:20.462782 8728 leader_election.cc:302] T 2560b1ec81ac4357b32c7a8ba9d72216 P 74077b61d6f54342bf9736b4a9f2eade [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74077b61d6f54342bf9736b4a9f2eade; no voters: 1f9cb72be98941b5a8e504cbe34132b9, a505e7cb133c4392bfda8a4a20874ae9
I20240501 07:59:20.462913 13396 raft_consensus.cc:2726] T 2560b1ec81ac4357b32c7a8ba9d72216 P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 07:59:20.463445 8728 leader_election.cc:334] T a93ce5841c7f40a9ab23f88efe7b85bb P 74077b61d6f54342bf9736b4a9f2eade [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111)
W20240501 07:59:20.463551 8728 leader_election.cc:334] T a93ce5841c7f40a9ab23f88efe7b85bb P 74077b61d6f54342bf9736b4a9f2eade [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111)
I20240501 07:59:20.463601 8728 leader_election.cc:302] T a93ce5841c7f40a9ab23f88efe7b85bb P 74077b61d6f54342bf9736b4a9f2eade [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74077b61d6f54342bf9736b4a9f2eade; no voters: 83f20e84af4c40d09a4804b6b26e697f, a505e7cb133c4392bfda8a4a20874ae9
I20240501 07:59:20.463734 13396 raft_consensus.cc:2726] T a93ce5841c7f40a9ab23f88efe7b85bb P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 07:59:20.529371 13424 raft_consensus.cc:482] T b7d57b7fd5234f50afbc859bc997f3e1 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 83f20e84af4c40d09a4804b6b26e697f)
I20240501 07:59:20.529528 13424 raft_consensus.cc:504] T b7d57b7fd5234f50afbc859bc997f3e1 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false } }
I20240501 07:59:20.529917 13424 leader_election.cc:288] T b7d57b7fd5234f50afbc859bc997f3e1 P 8b8b3ad3aa8f4053b2167dd550f2086f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133:39859), 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149)
I20240501 07:59:20.530326 8794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7d57b7fd5234f50afbc859bc997f3e1" candidate_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: false dest_uuid: "74077b61d6f54342bf9736b4a9f2eade" is_pre_election: true
I20240501 07:59:20.530503 8794 raft_consensus.cc:2454] T b7d57b7fd5234f50afbc859bc997f3e1 P 74077b61d6f54342bf9736b4a9f2eade [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b8b3ad3aa8f4053b2167dd550f2086f in term 2.
I20240501 07:59:20.530807 6248 leader_election.cc:302] T b7d57b7fd5234f50afbc859bc997f3e1 P 8b8b3ad3aa8f4053b2167dd550f2086f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74077b61d6f54342bf9736b4a9f2eade, 8b8b3ad3aa8f4053b2167dd550f2086f; no voters:
W20240501 07:59:20.530982 6249 leader_election.cc:334] T b7d57b7fd5234f50afbc859bc997f3e1 P 8b8b3ad3aa8f4053b2167dd550f2086f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111)
I20240501 07:59:20.531188 13424 raft_consensus.cc:2781] T b7d57b7fd5234f50afbc859bc997f3e1 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240501 07:59:20.531235 13424 raft_consensus.cc:482] T b7d57b7fd5234f50afbc859bc997f3e1 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 2 FOLLOWER]: Starting leader election (detected failure of leader 83f20e84af4c40d09a4804b6b26e697f)
I20240501 07:59:20.531283 13424 raft_consensus.cc:3037] T b7d57b7fd5234f50afbc859bc997f3e1 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 2 FOLLOWER]: Advancing to term 3
I20240501 07:59:20.532275 13424 raft_consensus.cc:504] T b7d57b7fd5234f50afbc859bc997f3e1 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 3 FOLLOWER]: Starting leader election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false } }
I20240501 07:59:20.532621 13424 leader_election.cc:288] T b7d57b7fd5234f50afbc859bc997f3e1 P 8b8b3ad3aa8f4053b2167dd550f2086f [CANDIDATE]: Term 3 election: Requested vote from peers 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133:39859), 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149)
I20240501 07:59:20.533208 8794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7d57b7fd5234f50afbc859bc997f3e1" candidate_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: false dest_uuid: "74077b61d6f54342bf9736b4a9f2eade"
I20240501 07:59:20.533317 8794 raft_consensus.cc:3037] T b7d57b7fd5234f50afbc859bc997f3e1 P 74077b61d6f54342bf9736b4a9f2eade [term 2 FOLLOWER]: Advancing to term 3
W20240501 07:59:20.533586 6249 leader_election.cc:334] T b7d57b7fd5234f50afbc859bc997f3e1 P 8b8b3ad3aa8f4053b2167dd550f2086f [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111)
I20240501 07:59:20.534327 8794 raft_consensus.cc:2454] T b7d57b7fd5234f50afbc859bc997f3e1 P 74077b61d6f54342bf9736b4a9f2eade [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b8b3ad3aa8f4053b2167dd550f2086f in term 3.
I20240501 07:59:20.534590 6248 leader_election.cc:302] T b7d57b7fd5234f50afbc859bc997f3e1 P 8b8b3ad3aa8f4053b2167dd550f2086f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 74077b61d6f54342bf9736b4a9f2eade, 8b8b3ad3aa8f4053b2167dd550f2086f; no voters: 83f20e84af4c40d09a4804b6b26e697f
I20240501 07:59:20.534756 13424 raft_consensus.cc:2781] T b7d57b7fd5234f50afbc859bc997f3e1 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 3 FOLLOWER]: Leader election won for term 3
I20240501 07:59:20.534854 13424 raft_consensus.cc:686] T b7d57b7fd5234f50afbc859bc997f3e1 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 3 LEADER]: Becoming Leader. State: Replica: 8b8b3ad3aa8f4053b2167dd550f2086f, State: Running, Role: LEADER
I20240501 07:59:20.534997 13424 consensus_queue.cc:237] T b7d57b7fd5234f50afbc859bc997f3e1 P 8b8b3ad3aa8f4053b2167dd550f2086f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 10, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false } }
I20240501 07:59:20.538154 6169 catalog_manager.cc:5484] T b7d57b7fd5234f50afbc859bc997f3e1 P 8b8b3ad3aa8f4053b2167dd550f2086f reported cstate change: term changed from 2 to 3, leader changed from 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136) to 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129). New cstate: current_term: 3 leader_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 07:59:20.551782 13424 raft_consensus.cc:482] T e5bd3a8f7a7640c09774f781009eb19c P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a505e7cb133c4392bfda8a4a20874ae9)
I20240501 07:59:20.551915 13424 raft_consensus.cc:504] T e5bd3a8f7a7640c09774f781009eb19c P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } }
I20240501 07:59:20.552297 13424 leader_election.cc:288] T e5bd3a8f7a7640c09774f781009eb19c P 8b8b3ad3aa8f4053b2167dd550f2086f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607), 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133:39859)
I20240501 07:59:20.552664 8794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5bd3a8f7a7640c09774f781009eb19c" candidate_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "74077b61d6f54342bf9736b4a9f2eade" is_pre_election: true
I20240501 07:59:20.552706 13418 raft_consensus.cc:482] T f788de4df5454d00ac9bb883f0abc9bd P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:20.552798 13418 raft_consensus.cc:504] T f788de4df5454d00ac9bb883f0abc9bd P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } }
I20240501 07:59:20.553221 13418 leader_election.cc:288] T f788de4df5454d00ac9bb883f0abc9bd P 13d05ebdca8c4f489c597ce550dd773d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131:34471), a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607)
W20240501 07:59:20.553347 6249 leader_election.cc:334] T e5bd3a8f7a7640c09774f781009eb19c P 8b8b3ad3aa8f4053b2167dd550f2086f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111)
I20240501 07:59:20.553429 6249 leader_election.cc:302] T e5bd3a8f7a7640c09774f781009eb19c P 8b8b3ad3aa8f4053b2167dd550f2086f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8b8b3ad3aa8f4053b2167dd550f2086f; no voters: 74077b61d6f54342bf9736b4a9f2eade, a505e7cb133c4392bfda8a4a20874ae9
I20240501 07:59:20.553731 13424 raft_consensus.cc:2726] T e5bd3a8f7a7640c09774f781009eb19c P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 07:59:20.554157 6575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f788de4df5454d00ac9bb883f0abc9bd" candidate_uuid: "13d05ebdca8c4f489c597ce550dd773d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" is_pre_election: true
I20240501 07:59:20.554309 6575 raft_consensus.cc:2454] T f788de4df5454d00ac9bb883f0abc9bd P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 13d05ebdca8c4f489c597ce550dd773d in term 0.
W20240501 07:59:20.554514 8859 leader_election.cc:334] T f788de4df5454d00ac9bb883f0abc9bd P 13d05ebdca8c4f489c597ce550dd773d [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111)
I20240501 07:59:20.554672 8861 leader_election.cc:302] T f788de4df5454d00ac9bb883f0abc9bd P 13d05ebdca8c4f489c597ce550dd773d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 13d05ebdca8c4f489c597ce550dd773d, 3b164cbbc00f4e8eb3254ddd349a3ff9; no voters: a505e7cb133c4392bfda8a4a20874ae9
I20240501 07:59:20.554823 13418 raft_consensus.cc:2781] T f788de4df5454d00ac9bb883f0abc9bd P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:20.554881 13418 raft_consensus.cc:482] T f788de4df5454d00ac9bb883f0abc9bd P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:20.554931 13418 raft_consensus.cc:3037] T f788de4df5454d00ac9bb883f0abc9bd P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:20.555742 13418 raft_consensus.cc:504] T f788de4df5454d00ac9bb883f0abc9bd P 13d05ebdca8c4f489c597ce550dd773d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } }
I20240501 07:59:20.556051 13418 leader_election.cc:288] T f788de4df5454d00ac9bb883f0abc9bd P 13d05ebdca8c4f489c597ce550dd773d [CANDIDATE]: Term 1 election: Requested vote from peers 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131:34471), a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607)
I20240501 07:59:20.556411 6575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f788de4df5454d00ac9bb883f0abc9bd" candidate_uuid: "13d05ebdca8c4f489c597ce550dd773d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9"
I20240501 07:59:20.556533 6575 raft_consensus.cc:3037] T f788de4df5454d00ac9bb883f0abc9bd P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Advancing to term 1
W20240501 07:59:20.557031 8859 leader_election.cc:334] T f788de4df5454d00ac9bb883f0abc9bd P 13d05ebdca8c4f489c597ce550dd773d [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111)
I20240501 07:59:20.557667 6575 raft_consensus.cc:2454] T f788de4df5454d00ac9bb883f0abc9bd P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13d05ebdca8c4f489c597ce550dd773d in term 1.
I20240501 07:59:20.560526 13942 raft_consensus.cc:482] T 2560b1ec81ac4357b32c7a8ba9d72216 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a505e7cb133c4392bfda8a4a20874ae9)
I20240501 07:59:20.560647 13942 raft_consensus.cc:504] T 2560b1ec81ac4357b32c7a8ba9d72216 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: true } }
I20240501 07:59:20.561232 8861 leader_election.cc:302] T f788de4df5454d00ac9bb883f0abc9bd P 13d05ebdca8c4f489c597ce550dd773d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 13d05ebdca8c4f489c597ce550dd773d, 3b164cbbc00f4e8eb3254ddd349a3ff9; no voters: a505e7cb133c4392bfda8a4a20874ae9
I20240501 07:59:20.561415 13418 raft_consensus.cc:2781] T f788de4df5454d00ac9bb883f0abc9bd P 13d05ebdca8c4f489c597ce550dd773d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:20.561506 13418 raft_consensus.cc:686] T f788de4df5454d00ac9bb883f0abc9bd P 13d05ebdca8c4f489c597ce550dd773d [term 1 LEADER]: Becoming Leader. State: Replica: 13d05ebdca8c4f489c597ce550dd773d, State: Running, Role: LEADER
I20240501 07:59:20.561650 13418 consensus_queue.cc:237] T f788de4df5454d00ac9bb883f0abc9bd P 13d05ebdca8c4f489c597ce550dd773d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } }
I20240501 07:59:20.562196 13942 leader_election.cc:288] T 2560b1ec81ac4357b32c7a8ba9d72216 P 1f9cb72be98941b5a8e504cbe34132b9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133:39859), a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607)
I20240501 07:59:20.562387 8794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2560b1ec81ac4357b32c7a8ba9d72216" candidate_uuid: "1f9cb72be98941b5a8e504cbe34132b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "74077b61d6f54342bf9736b4a9f2eade" is_pre_election: true
I20240501 07:59:20.562538 8794 raft_consensus.cc:2454] T 2560b1ec81ac4357b32c7a8ba9d72216 P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f9cb72be98941b5a8e504cbe34132b9 in term 1.
I20240501 07:59:20.562813 6641 leader_election.cc:302] T 2560b1ec81ac4357b32c7a8ba9d72216 P 1f9cb72be98941b5a8e504cbe34132b9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f9cb72be98941b5a8e504cbe34132b9, 74077b61d6f54342bf9736b4a9f2eade; no voters:
W20240501 07:59:20.562979 6642 leader_election.cc:334] T 2560b1ec81ac4357b32c7a8ba9d72216 P 1f9cb72be98941b5a8e504cbe34132b9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111)
I20240501 07:59:20.562979 13942 raft_consensus.cc:2781] T 2560b1ec81ac4357b32c7a8ba9d72216 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 07:59:20.563098 13942 raft_consensus.cc:482] T 2560b1ec81ac4357b32c7a8ba9d72216 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Starting leader election (detected failure of leader a505e7cb133c4392bfda8a4a20874ae9)
I20240501 07:59:20.563141 13942 raft_consensus.cc:3037] T 2560b1ec81ac4357b32c7a8ba9d72216 P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:20.563730 13418 raft_consensus.cc:482] T f88fe033bce14bf0b1b63a52a7ba137e P 13d05ebdca8c4f489c597ce550dd773d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a505e7cb133c4392bfda8a4a20874ae9)
I20240501 07:59:20.563831 13418 raft_consensus.cc:504] T f88fe033bce14bf0b1b63a52a7ba137e P 13d05ebdca8c4f489c597ce550dd773d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { promote: false } }
I20240501 07:59:20.564023 6169 catalog_manager.cc:5484] T f788de4df5454d00ac9bb883f0abc9bd P 13d05ebdca8c4f489c597ce550dd773d reported cstate change: term changed from 0 to 1, leader changed from <none> to 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134). New cstate: current_term: 1 leader_uuid: "13d05ebdca8c4f489c597ce550dd773d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } health_report { overall_health: UNKNOWN } } }
I20240501 07:59:20.564159 13942 raft_consensus.cc:504] T 2560b1ec81ac4357b32c7a8ba9d72216 P 1f9cb72be98941b5a8e504cbe34132b9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: true } }
I20240501 07:59:20.564347 13418 leader_election.cc:288] T f88fe033bce14bf0b1b63a52a7ba137e P 13d05ebdca8c4f489c597ce550dd773d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607), 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130:32941)
I20240501 07:59:20.564548 13942 leader_election.cc:288] T 2560b1ec81ac4357b32c7a8ba9d72216 P 1f9cb72be98941b5a8e504cbe34132b9 [CANDIDATE]: Term 2 election: Requested vote from peers 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133:39859), a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607)
I20240501 07:59:20.564883 8794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2560b1ec81ac4357b32c7a8ba9d72216" candidate_uuid: "1f9cb72be98941b5a8e504cbe34132b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "74077b61d6f54342bf9736b4a9f2eade"
I20240501 07:59:20.565029 8794 raft_consensus.cc:3037] T 2560b1ec81ac4357b32c7a8ba9d72216 P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:20.565097 6444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f88fe033bce14bf0b1b63a52a7ba137e" candidate_uuid: "13d05ebdca8c4f489c597ce550dd773d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8315d22c56f7439a9e1fc82f153f018c" is_pre_election: true
I20240501 07:59:20.566170 8794 raft_consensus.cc:2454] T 2560b1ec81ac4357b32c7a8ba9d72216 P 74077b61d6f54342bf9736b4a9f2eade [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f9cb72be98941b5a8e504cbe34132b9 in term 2.
I20240501 07:59:20.566524 6641 leader_election.cc:302] T 2560b1ec81ac4357b32c7a8ba9d72216 P 1f9cb72be98941b5a8e504cbe34132b9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f9cb72be98941b5a8e504cbe34132b9, 74077b61d6f54342bf9736b4a9f2eade; no voters:
I20240501 07:59:20.566709 14067 raft_consensus.cc:2781] T 2560b1ec81ac4357b32c7a8ba9d72216 P 1f9cb72be98941b5a8e504cbe34132b9 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 07:59:20.566793 14067 raft_consensus.cc:686] T 2560b1ec81ac4357b32c7a8ba9d72216 P 1f9cb72be98941b5a8e504cbe34132b9 [term 2 LEADER]: Becoming Leader. State: Replica: 1f9cb72be98941b5a8e504cbe34132b9, State: Running, Role: LEADER
I20240501 07:59:20.566944 14067 consensus_queue.cc:237] T 2560b1ec81ac4357b32c7a8ba9d72216 P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: true } }
W20240501 07:59:20.568097 6642 leader_election.cc:334] T 2560b1ec81ac4357b32c7a8ba9d72216 P 1f9cb72be98941b5a8e504cbe34132b9 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111)
W20240501 07:59:20.568300 8859 leader_election.cc:334] T f88fe033bce14bf0b1b63a52a7ba137e P 13d05ebdca8c4f489c597ce550dd773d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111)
I20240501 07:59:20.568370 8859 leader_election.cc:302] T f88fe033bce14bf0b1b63a52a7ba137e P 13d05ebdca8c4f489c597ce550dd773d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 13d05ebdca8c4f489c597ce550dd773d; no voters: 8315d22c56f7439a9e1fc82f153f018c, a505e7cb133c4392bfda8a4a20874ae9
I20240501 07:59:20.568504 13418 raft_consensus.cc:2726] T f88fe033bce14bf0b1b63a52a7ba137e P 13d05ebdca8c4f489c597ce550dd773d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 07:59:20.570132 6165 catalog_manager.cc:5484] T 2560b1ec81ac4357b32c7a8ba9d72216 P 1f9cb72be98941b5a8e504cbe34132b9 reported cstate change: term changed from 1 to 2, leader changed from a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135) to 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132). New cstate: current_term: 2 leader_uuid: "1f9cb72be98941b5a8e504cbe34132b9" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 07:59:20.571187 13396 raft_consensus.cc:482] T c16b7962c18d4e02b850c7c7c678eae4 P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a505e7cb133c4392bfda8a4a20874ae9)
I20240501 07:59:20.571293 13396 raft_consensus.cc:504] T c16b7962c18d4e02b850c7c7c678eae4 P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { promote: true } }
I20240501 07:59:20.571714 13396 leader_election.cc:288] T c16b7962c18d4e02b850c7c7c678eae4 P 74077b61d6f54342bf9736b4a9f2eade [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134:40141), a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607)
I20240501 07:59:20.572829 8924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c16b7962c18d4e02b850c7c7c678eae4" candidate_uuid: "74077b61d6f54342bf9736b4a9f2eade" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "13d05ebdca8c4f489c597ce550dd773d" is_pre_election: true
W20240501 07:59:20.573194 8728 leader_election.cc:334] T c16b7962c18d4e02b850c7c7c678eae4 P 74077b61d6f54342bf9736b4a9f2eade [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111)
I20240501 07:59:20.573863 8729 leader_election.cc:302] T c16b7962c18d4e02b850c7c7c678eae4 P 74077b61d6f54342bf9736b4a9f2eade [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74077b61d6f54342bf9736b4a9f2eade; no voters: 13d05ebdca8c4f489c597ce550dd773d, a505e7cb133c4392bfda8a4a20874ae9
I20240501 07:59:20.574065 13396 raft_consensus.cc:2726] T c16b7962c18d4e02b850c7c7c678eae4 P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 07:59:20.576412 13396 raft_consensus.cc:482] T af718dc43add4edfac703013655699f4 P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a505e7cb133c4392bfda8a4a20874ae9)
I20240501 07:59:20.576511 13396 raft_consensus.cc:504] T af718dc43add4edfac703013655699f4 P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } }
I20240501 07:59:20.576896 13396 leader_election.cc:288] T af718dc43add4edfac703013655699f4 P 74077b61d6f54342bf9736b4a9f2eade [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607), 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131:34471)
I20240501 07:59:20.577318 6575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af718dc43add4edfac703013655699f4" candidate_uuid: "74077b61d6f54342bf9736b4a9f2eade" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" is_pre_election: true
W20240501 07:59:20.578115 8728 leader_election.cc:334] T af718dc43add4edfac703013655699f4 P 74077b61d6f54342bf9736b4a9f2eade [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111)
I20240501 07:59:20.578174 8728 leader_election.cc:302] T af718dc43add4edfac703013655699f4 P 74077b61d6f54342bf9736b4a9f2eade [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74077b61d6f54342bf9736b4a9f2eade; no voters: 3b164cbbc00f4e8eb3254ddd349a3ff9, a505e7cb133c4392bfda8a4a20874ae9
I20240501 07:59:20.578297 13396 raft_consensus.cc:2726] T af718dc43add4edfac703013655699f4 P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 07:59:20.581020 11107 raft_consensus.cc:482] T af718dc43add4edfac703013655699f4 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a505e7cb133c4392bfda8a4a20874ae9)
I20240501 07:59:20.581110 11107 raft_consensus.cc:504] T af718dc43add4edfac703013655699f4 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } }
I20240501 07:59:20.581480 11107 leader_election.cc:288] T af718dc43add4edfac703013655699f4 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133:39859), a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607)
I20240501 07:59:20.581841 8794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af718dc43add4edfac703013655699f4" candidate_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "74077b61d6f54342bf9736b4a9f2eade" is_pre_election: true
I20240501 07:59:20.581954 8794 raft_consensus.cc:2454] T af718dc43add4edfac703013655699f4 P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b164cbbc00f4e8eb3254ddd349a3ff9 in term 1.
I20240501 07:59:20.582407 6510 leader_election.cc:302] T af718dc43add4edfac703013655699f4 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b164cbbc00f4e8eb3254ddd349a3ff9, 74077b61d6f54342bf9736b4a9f2eade; no voters:
I20240501 07:59:20.582686 11107 raft_consensus.cc:2781] T af718dc43add4edfac703013655699f4 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 07:59:20.582742 11107 raft_consensus.cc:482] T af718dc43add4edfac703013655699f4 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Starting leader election (detected failure of leader a505e7cb133c4392bfda8a4a20874ae9)
I20240501 07:59:20.582778 11107 raft_consensus.cc:3037] T af718dc43add4edfac703013655699f4 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Advancing to term 2
W20240501 07:59:20.582753 6511 leader_election.cc:334] T af718dc43add4edfac703013655699f4 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111)
I20240501 07:59:20.583842 11107 raft_consensus.cc:504] T af718dc43add4edfac703013655699f4 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } }
I20240501 07:59:20.584110 11107 leader_election.cc:288] T af718dc43add4edfac703013655699f4 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [CANDIDATE]: Term 2 election: Requested vote from peers 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133:39859), a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607)
I20240501 07:59:20.584549 8794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af718dc43add4edfac703013655699f4" candidate_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "74077b61d6f54342bf9736b4a9f2eade"
I20240501 07:59:20.584692 8794 raft_consensus.cc:3037] T af718dc43add4edfac703013655699f4 P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:20.584992 13961 raft_consensus.cc:482] T c86c420310ec42ffa5b24304fba74267 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83f20e84af4c40d09a4804b6b26e697f)
W20240501 07:59:20.585070 6511 leader_election.cc:334] T af718dc43add4edfac703013655699f4 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111)
I20240501 07:59:20.585120 13961 raft_consensus.cc:504] T c86c420310ec42ffa5b24304fba74267 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { replace: true } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: true } }
I20240501 07:59:20.585520 13961 leader_election.cc:288] T c86c420310ec42ffa5b24304fba74267 P 8315d22c56f7439a9e1fc82f153f018c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149), a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607)
I20240501 07:59:20.585764 8794 raft_consensus.cc:2454] T af718dc43add4edfac703013655699f4 P 74077b61d6f54342bf9736b4a9f2eade [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b164cbbc00f4e8eb3254ddd349a3ff9 in term 2.
I20240501 07:59:20.586014 6510 leader_election.cc:302] T af718dc43add4edfac703013655699f4 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b164cbbc00f4e8eb3254ddd349a3ff9, 74077b61d6f54342bf9736b4a9f2eade; no voters: a505e7cb133c4392bfda8a4a20874ae9
I20240501 07:59:20.586421 11107 raft_consensus.cc:2781] T af718dc43add4edfac703013655699f4 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 07:59:20.586506 11107 raft_consensus.cc:686] T af718dc43add4edfac703013655699f4 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 2 LEADER]: Becoming Leader. State: Replica: 3b164cbbc00f4e8eb3254ddd349a3ff9, State: Running, Role: LEADER
W20240501 07:59:20.586604 6380 leader_election.cc:334] T c86c420310ec42ffa5b24304fba74267 P 8315d22c56f7439a9e1fc82f153f018c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111)
I20240501 07:59:20.586647 11107 consensus_queue.cc:237] T af718dc43add4edfac703013655699f4 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } }
W20240501 07:59:20.586724 6380 leader_election.cc:334] T c86c420310ec42ffa5b24304fba74267 P 8315d22c56f7439a9e1fc82f153f018c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111)
I20240501 07:59:20.586764 6380 leader_election.cc:302] T c86c420310ec42ffa5b24304fba74267 P 8315d22c56f7439a9e1fc82f153f018c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8315d22c56f7439a9e1fc82f153f018c; no voters: 83f20e84af4c40d09a4804b6b26e697f, a505e7cb133c4392bfda8a4a20874ae9
I20240501 07:59:20.586872 13961 raft_consensus.cc:2726] T c86c420310ec42ffa5b24304fba74267 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 07:59:20.588734 6165 catalog_manager.cc:5484] T af718dc43add4edfac703013655699f4 P 3b164cbbc00f4e8eb3254ddd349a3ff9 reported cstate change: term changed from 1 to 2, leader changed from a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135) to 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131). New cstate: current_term: 2 leader_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } health_report { overall_health: HEALTHY } } }
I20240501 07:59:20.590812 13418 raft_consensus.cc:482] T c16b7962c18d4e02b850c7c7c678eae4 P 13d05ebdca8c4f489c597ce550dd773d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a505e7cb133c4392bfda8a4a20874ae9)
I20240501 07:59:20.590920 13418 raft_consensus.cc:504] T c16b7962c18d4e02b850c7c7c678eae4 P 13d05ebdca8c4f489c597ce550dd773d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { promote: true } }
I20240501 07:59:20.591272 13418 leader_election.cc:288] T c16b7962c18d4e02b850c7c7c678eae4 P 13d05ebdca8c4f489c597ce550dd773d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133:39859), a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607)
I20240501 07:59:20.591615 8794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c16b7962c18d4e02b850c7c7c678eae4" candidate_uuid: "13d05ebdca8c4f489c597ce550dd773d" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "74077b61d6f54342bf9736b4a9f2eade" is_pre_election: true
I20240501 07:59:20.591746 8794 raft_consensus.cc:2454] T c16b7962c18d4e02b850c7c7c678eae4 P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 13d05ebdca8c4f489c597ce550dd773d in term 1.
I20240501 07:59:20.592048 8858 leader_election.cc:302] T c16b7962c18d4e02b850c7c7c678eae4 P 13d05ebdca8c4f489c597ce550dd773d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13d05ebdca8c4f489c597ce550dd773d, 74077b61d6f54342bf9736b4a9f2eade; no voters:
I20240501 07:59:20.592213 13418 raft_consensus.cc:2781] T c16b7962c18d4e02b850c7c7c678eae4 P 13d05ebdca8c4f489c597ce550dd773d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 07:59:20.592254 13418 raft_consensus.cc:482] T c16b7962c18d4e02b850c7c7c678eae4 P 13d05ebdca8c4f489c597ce550dd773d [term 1 FOLLOWER]: Starting leader election (detected failure of leader a505e7cb133c4392bfda8a4a20874ae9)
I20240501 07:59:20.592285 13418 raft_consensus.cc:3037] T c16b7962c18d4e02b850c7c7c678eae4 P 13d05ebdca8c4f489c597ce550dd773d [term 1 FOLLOWER]: Advancing to term 2
W20240501 07:59:20.592320 8859 leader_election.cc:334] T c16b7962c18d4e02b850c7c7c678eae4 P 13d05ebdca8c4f489c597ce550dd773d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111)
I20240501 07:59:20.593369 13418 raft_consensus.cc:504] T c16b7962c18d4e02b850c7c7c678eae4 P 13d05ebdca8c4f489c597ce550dd773d [term 2 FOLLOWER]: Starting leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { promote: true } }
I20240501 07:59:20.593943 13418 leader_election.cc:288] T c16b7962c18d4e02b850c7c7c678eae4 P 13d05ebdca8c4f489c597ce550dd773d [CANDIDATE]: Term 2 election: Requested vote from peers 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133:39859), a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607)
I20240501 07:59:20.593984 8794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c16b7962c18d4e02b850c7c7c678eae4" candidate_uuid: "13d05ebdca8c4f489c597ce550dd773d" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "74077b61d6f54342bf9736b4a9f2eade"
I20240501 07:59:20.594106 8794 raft_consensus.cc:3037] T c16b7962c18d4e02b850c7c7c678eae4 P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Advancing to term 2
W20240501 07:59:20.594693 8859 leader_election.cc:334] T c16b7962c18d4e02b850c7c7c678eae4 P 13d05ebdca8c4f489c597ce550dd773d [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111)
I20240501 07:59:20.594998 8794 raft_consensus.cc:2454] T c16b7962c18d4e02b850c7c7c678eae4 P 74077b61d6f54342bf9736b4a9f2eade [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13d05ebdca8c4f489c597ce550dd773d in term 2.
I20240501 07:59:20.595255 8858 leader_election.cc:302] T c16b7962c18d4e02b850c7c7c678eae4 P 13d05ebdca8c4f489c597ce550dd773d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 13d05ebdca8c4f489c597ce550dd773d, 74077b61d6f54342bf9736b4a9f2eade; no voters: a505e7cb133c4392bfda8a4a20874ae9
I20240501 07:59:20.595417 13040 raft_consensus.cc:2781] T c16b7962c18d4e02b850c7c7c678eae4 P 13d05ebdca8c4f489c597ce550dd773d [term 2 FOLLOWER]: Leader election won for term 2
I20240501 07:59:20.595500 13040 raft_consensus.cc:686] T c16b7962c18d4e02b850c7c7c678eae4 P 13d05ebdca8c4f489c597ce550dd773d [term 2 LEADER]: Becoming Leader. State: Replica: 13d05ebdca8c4f489c597ce550dd773d, State: Running, Role: LEADER
I20240501 07:59:20.595634 13040 consensus_queue.cc:237] T c16b7962c18d4e02b850c7c7c678eae4 P 13d05ebdca8c4f489c597ce550dd773d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { promote: true } }
I20240501 07:59:20.598228 6165 catalog_manager.cc:5484] T c16b7962c18d4e02b850c7c7c678eae4 P 13d05ebdca8c4f489c597ce550dd773d reported cstate change: term changed from 1 to 2, leader changed from a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135) to 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134). New cstate: current_term: 2 leader_uuid: "13d05ebdca8c4f489c597ce550dd773d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240501 07:59:20.609140 6511 consensus_peers.cc:462] T 0b24f67cf1a64debb23ee6618c820d33 P 3b164cbbc00f4e8eb3254ddd349a3ff9 -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111)
I20240501 07:59:20.624671 13396 raft_consensus.cc:482] T e5bd3a8f7a7640c09774f781009eb19c P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a505e7cb133c4392bfda8a4a20874ae9)
I20240501 07:59:20.624828 13396 raft_consensus.cc:504] T e5bd3a8f7a7640c09774f781009eb19c P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } }
I20240501 07:59:20.625423 13396 leader_election.cc:288] T e5bd3a8f7a7640c09774f781009eb19c P 74077b61d6f54342bf9736b4a9f2eade [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129:46875), a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607)
I20240501 07:59:20.625895 6311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5bd3a8f7a7640c09774f781009eb19c" candidate_uuid: "74077b61d6f54342bf9736b4a9f2eade" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" is_pre_election: true
I20240501 07:59:20.626040 6311 raft_consensus.cc:2454] T e5bd3a8f7a7640c09774f781009eb19c P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74077b61d6f54342bf9736b4a9f2eade in term 1.
I20240501 07:59:20.626405 8727 leader_election.cc:302] T e5bd3a8f7a7640c09774f781009eb19c P 74077b61d6f54342bf9736b4a9f2eade [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74077b61d6f54342bf9736b4a9f2eade, 8b8b3ad3aa8f4053b2167dd550f2086f; no voters:
I20240501 07:59:20.626601 13396 raft_consensus.cc:2781] T e5bd3a8f7a7640c09774f781009eb19c P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 07:59:20.626677 13396 raft_consensus.cc:482] T e5bd3a8f7a7640c09774f781009eb19c P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Starting leader election (detected failure of leader a505e7cb133c4392bfda8a4a20874ae9)
I20240501 07:59:20.626726 13396 raft_consensus.cc:3037] T e5bd3a8f7a7640c09774f781009eb19c P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Advancing to term 2
W20240501 07:59:20.627211 8728 leader_election.cc:334] T e5bd3a8f7a7640c09774f781009eb19c P 74077b61d6f54342bf9736b4a9f2eade [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111)
I20240501 07:59:20.627763 13396 raft_consensus.cc:504] T e5bd3a8f7a7640c09774f781009eb19c P 74077b61d6f54342bf9736b4a9f2eade [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } }
I20240501 07:59:20.628108 13396 leader_election.cc:288] T e5bd3a8f7a7640c09774f781009eb19c P 74077b61d6f54342bf9736b4a9f2eade [CANDIDATE]: Term 2 election: Requested vote from peers 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129:46875), a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607)
I20240501 07:59:20.628458 6311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5bd3a8f7a7640c09774f781009eb19c" candidate_uuid: "74077b61d6f54342bf9736b4a9f2eade" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f"
I20240501 07:59:20.628583 6311 raft_consensus.cc:3037] T e5bd3a8f7a7640c09774f781009eb19c P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:20.629702 6311 raft_consensus.cc:2454] T e5bd3a8f7a7640c09774f781009eb19c P 8b8b3ad3aa8f4053b2167dd550f2086f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74077b61d6f54342bf9736b4a9f2eade in term 2.
W20240501 07:59:20.629801 8728 leader_election.cc:334] T e5bd3a8f7a7640c09774f781009eb19c P 74077b61d6f54342bf9736b4a9f2eade [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111)
I20240501 07:59:20.630015 8727 leader_election.cc:302] T e5bd3a8f7a7640c09774f781009eb19c P 74077b61d6f54342bf9736b4a9f2eade [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 74077b61d6f54342bf9736b4a9f2eade, 8b8b3ad3aa8f4053b2167dd550f2086f; no voters: a505e7cb133c4392bfda8a4a20874ae9
I20240501 07:59:20.630234 13396 raft_consensus.cc:2781] T e5bd3a8f7a7640c09774f781009eb19c P 74077b61d6f54342bf9736b4a9f2eade [term 2 FOLLOWER]: Leader election won for term 2
I20240501 07:59:20.630316 13396 raft_consensus.cc:686] T e5bd3a8f7a7640c09774f781009eb19c P 74077b61d6f54342bf9736b4a9f2eade [term 2 LEADER]: Becoming Leader. State: Replica: 74077b61d6f54342bf9736b4a9f2eade, State: Running, Role: LEADER
I20240501 07:59:20.630463 13396 consensus_queue.cc:237] T e5bd3a8f7a7640c09774f781009eb19c P 74077b61d6f54342bf9736b4a9f2eade [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } }
I20240501 07:59:20.633817 6165 catalog_manager.cc:5484] T e5bd3a8f7a7640c09774f781009eb19c P 74077b61d6f54342bf9736b4a9f2eade reported cstate change: term changed from 1 to 2, leader changed from a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135) to 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133). New cstate: current_term: 2 leader_uuid: "74077b61d6f54342bf9736b4a9f2eade" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } health_report { overall_health: HEALTHY } } }
I20240501 07:59:20.636590 13961 raft_consensus.cc:482] T f88fe033bce14bf0b1b63a52a7ba137e P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a505e7cb133c4392bfda8a4a20874ae9)
I20240501 07:59:20.636701 13961 raft_consensus.cc:504] T f88fe033bce14bf0b1b63a52a7ba137e P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { promote: false } }
I20240501 07:59:20.637090 13961 leader_election.cc:288] T f88fe033bce14bf0b1b63a52a7ba137e P 8315d22c56f7439a9e1fc82f153f018c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134:40141), a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607)
I20240501 07:59:20.637485 8924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f88fe033bce14bf0b1b63a52a7ba137e" candidate_uuid: "8315d22c56f7439a9e1fc82f153f018c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "13d05ebdca8c4f489c597ce550dd773d" is_pre_election: true
I20240501 07:59:20.637629 8924 raft_consensus.cc:2454] T f88fe033bce14bf0b1b63a52a7ba137e P 13d05ebdca8c4f489c597ce550dd773d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8315d22c56f7439a9e1fc82f153f018c in term 1.
I20240501 07:59:20.637933 6381 leader_election.cc:302] T f88fe033bce14bf0b1b63a52a7ba137e P 8315d22c56f7439a9e1fc82f153f018c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13d05ebdca8c4f489c597ce550dd773d, 8315d22c56f7439a9e1fc82f153f018c; no voters:
I20240501 07:59:20.638078 13961 raft_consensus.cc:2781] T f88fe033bce14bf0b1b63a52a7ba137e P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 07:59:20.638130 13961 raft_consensus.cc:482] T f88fe033bce14bf0b1b63a52a7ba137e P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Starting leader election (detected failure of leader a505e7cb133c4392bfda8a4a20874ae9)
I20240501 07:59:20.638162 13961 raft_consensus.cc:3037] T f88fe033bce14bf0b1b63a52a7ba137e P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Advancing to term 2
W20240501 07:59:20.638371 6380 leader_election.cc:334] T f88fe033bce14bf0b1b63a52a7ba137e P 8315d22c56f7439a9e1fc82f153f018c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111)
I20240501 07:59:20.639210 13961 raft_consensus.cc:504] T f88fe033bce14bf0b1b63a52a7ba137e P 8315d22c56f7439a9e1fc82f153f018c [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { promote: false } }
I20240501 07:59:20.639525 13961 leader_election.cc:288] T f88fe033bce14bf0b1b63a52a7ba137e P 8315d22c56f7439a9e1fc82f153f018c [CANDIDATE]: Term 2 election: Requested vote from peers 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134:40141), a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607)
I20240501 07:59:20.639837 8924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f88fe033bce14bf0b1b63a52a7ba137e" candidate_uuid: "8315d22c56f7439a9e1fc82f153f018c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "13d05ebdca8c4f489c597ce550dd773d"
I20240501 07:59:20.639966 8924 raft_consensus.cc:3037] T f88fe033bce14bf0b1b63a52a7ba137e P 13d05ebdca8c4f489c597ce550dd773d [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:20.640748 8924 raft_consensus.cc:2454] T f88fe033bce14bf0b1b63a52a7ba137e P 13d05ebdca8c4f489c597ce550dd773d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8315d22c56f7439a9e1fc82f153f018c in term 2.
I20240501 07:59:20.641072 6381 leader_election.cc:302] T f88fe033bce14bf0b1b63a52a7ba137e P 8315d22c56f7439a9e1fc82f153f018c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13d05ebdca8c4f489c597ce550dd773d, 8315d22c56f7439a9e1fc82f153f018c; no voters:
I20240501 07:59:20.641244 13961 raft_consensus.cc:2781] T f88fe033bce14bf0b1b63a52a7ba137e P 8315d22c56f7439a9e1fc82f153f018c [term 2 FOLLOWER]: Leader election won for term 2
I20240501 07:59:20.641315 13961 raft_consensus.cc:686] T f88fe033bce14bf0b1b63a52a7ba137e P 8315d22c56f7439a9e1fc82f153f018c [term 2 LEADER]: Becoming Leader. State: Replica: 8315d22c56f7439a9e1fc82f153f018c, State: Running, Role: LEADER
I20240501 07:59:20.641429 13961 consensus_queue.cc:237] T f88fe033bce14bf0b1b63a52a7ba137e P 8315d22c56f7439a9e1fc82f153f018c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { promote: false } }
W20240501 07:59:20.641889 6380 leader_election.cc:334] T f88fe033bce14bf0b1b63a52a7ba137e P 8315d22c56f7439a9e1fc82f153f018c [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111)
I20240501 07:59:20.644343 6165 catalog_manager.cc:5484] T f88fe033bce14bf0b1b63a52a7ba137e P 8315d22c56f7439a9e1fc82f153f018c reported cstate change: term changed from 1 to 2, leader changed from a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135) to 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130). New cstate: current_term: 2 leader_uuid: "8315d22c56f7439a9e1fc82f153f018c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 07:59:20.659214 13961 raft_consensus.cc:482] T 8f36eefcda3f475d9571e92fdeb52eb4 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a505e7cb133c4392bfda8a4a20874ae9)
I20240501 07:59:20.659337 13961 raft_consensus.cc:504] T 8f36eefcda3f475d9571e92fdeb52eb4 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { replace: true } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: true } }
I20240501 07:59:20.659794 13961 leader_election.cc:288] T 8f36eefcda3f475d9571e92fdeb52eb4 P 8315d22c56f7439a9e1fc82f153f018c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607), 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134:40141)
I20240501 07:59:20.660233 8924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f36eefcda3f475d9571e92fdeb52eb4" candidate_uuid: "8315d22c56f7439a9e1fc82f153f018c" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "13d05ebdca8c4f489c597ce550dd773d" is_pre_election: true
I20240501 07:59:20.660306 13961 raft_consensus.cc:482] T 72aa4dc7664b48b7a73d34feba7d44d3 P 8315d22c56f7439a9e1fc82f153f018c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a505e7cb133c4392bfda8a4a20874ae9)
I20240501 07:59:20.660387 13961 raft_consensus.cc:504] T 72aa4dc7664b48b7a73d34feba7d44d3 P 8315d22c56f7439a9e1fc82f153f018c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { promote: false } }
I20240501 07:59:20.660739 13961 leader_election.cc:288] T 72aa4dc7664b48b7a73d34feba7d44d3 P 8315d22c56f7439a9e1fc82f153f018c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134:40141), a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607)
I20240501 07:59:20.661068 8924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72aa4dc7664b48b7a73d34feba7d44d3" candidate_uuid: "8315d22c56f7439a9e1fc82f153f018c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "13d05ebdca8c4f489c597ce550dd773d" is_pre_election: true
I20240501 07:59:20.661196 8924 raft_consensus.cc:2454] T 72aa4dc7664b48b7a73d34feba7d44d3 P 13d05ebdca8c4f489c597ce550dd773d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8315d22c56f7439a9e1fc82f153f018c in term 2.
W20240501 07:59:20.661187 6380 leader_election.cc:334] T 8f36eefcda3f475d9571e92fdeb52eb4 P 8315d22c56f7439a9e1fc82f153f018c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111)
I20240501 07:59:20.661260 6380 leader_election.cc:302] T 8f36eefcda3f475d9571e92fdeb52eb4 P 8315d22c56f7439a9e1fc82f153f018c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8315d22c56f7439a9e1fc82f153f018c; no voters: 13d05ebdca8c4f489c597ce550dd773d, a505e7cb133c4392bfda8a4a20874ae9
I20240501 07:59:20.661474 6381 leader_election.cc:302] T 72aa4dc7664b48b7a73d34feba7d44d3 P 8315d22c56f7439a9e1fc82f153f018c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13d05ebdca8c4f489c597ce550dd773d, 8315d22c56f7439a9e1fc82f153f018c; no voters:
I20240501 07:59:20.661646 13172 raft_consensus.cc:2726] T 8f36eefcda3f475d9571e92fdeb52eb4 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 07:59:20.661685 13961 raft_consensus.cc:2781] T 72aa4dc7664b48b7a73d34feba7d44d3 P 8315d22c56f7439a9e1fc82f153f018c [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240501 07:59:20.661721 13961 raft_consensus.cc:482] T 72aa4dc7664b48b7a73d34feba7d44d3 P 8315d22c56f7439a9e1fc82f153f018c [term 2 FOLLOWER]: Starting leader election (detected failure of leader a505e7cb133c4392bfda8a4a20874ae9)
I20240501 07:59:20.661773 13961 raft_consensus.cc:3037] T 72aa4dc7664b48b7a73d34feba7d44d3 P 8315d22c56f7439a9e1fc82f153f018c [term 2 FOLLOWER]: Advancing to term 3
I20240501 07:59:20.662788 13961 raft_consensus.cc:504] T 72aa4dc7664b48b7a73d34feba7d44d3 P 8315d22c56f7439a9e1fc82f153f018c [term 3 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { promote: false } }
I20240501 07:59:20.663539 13961 leader_election.cc:288] T 72aa4dc7664b48b7a73d34feba7d44d3 P 8315d22c56f7439a9e1fc82f153f018c [CANDIDATE]: Term 3 election: Requested vote from peers 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134:40141), a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607)
I20240501 07:59:20.663938 8924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72aa4dc7664b48b7a73d34feba7d44d3" candidate_uuid: "8315d22c56f7439a9e1fc82f153f018c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "13d05ebdca8c4f489c597ce550dd773d"
I20240501 07:59:20.664048 8924 raft_consensus.cc:3037] T 72aa4dc7664b48b7a73d34feba7d44d3 P 13d05ebdca8c4f489c597ce550dd773d [term 2 FOLLOWER]: Advancing to term 3
W20240501 07:59:20.665374 6380 leader_election.cc:334] T 72aa4dc7664b48b7a73d34feba7d44d3 P 8315d22c56f7439a9e1fc82f153f018c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111)
I20240501 07:59:20.665668 8924 raft_consensus.cc:2454] T 72aa4dc7664b48b7a73d34feba7d44d3 P 13d05ebdca8c4f489c597ce550dd773d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8315d22c56f7439a9e1fc82f153f018c in term 3.
I20240501 07:59:20.665946 6381 leader_election.cc:302] T 72aa4dc7664b48b7a73d34feba7d44d3 P 8315d22c56f7439a9e1fc82f153f018c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13d05ebdca8c4f489c597ce550dd773d, 8315d22c56f7439a9e1fc82f153f018c; no voters:
I20240501 07:59:20.666252 13961 raft_consensus.cc:2781] T 72aa4dc7664b48b7a73d34feba7d44d3 P 8315d22c56f7439a9e1fc82f153f018c [term 3 FOLLOWER]: Leader election won for term 3
I20240501 07:59:20.666378 13961 raft_consensus.cc:686] T 72aa4dc7664b48b7a73d34feba7d44d3 P 8315d22c56f7439a9e1fc82f153f018c [term 3 LEADER]: Becoming Leader. State: Replica: 8315d22c56f7439a9e1fc82f153f018c, State: Running, Role: LEADER
I20240501 07:59:20.666720 13961 consensus_queue.cc:237] T 72aa4dc7664b48b7a73d34feba7d44d3 P 8315d22c56f7439a9e1fc82f153f018c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { promote: false } }
I20240501 07:59:20.669179 6165 catalog_manager.cc:5484] T 72aa4dc7664b48b7a73d34feba7d44d3 P 8315d22c56f7439a9e1fc82f153f018c reported cstate change: term changed from 2 to 3, leader changed from a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135) to 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130). New cstate: current_term: 3 leader_uuid: "8315d22c56f7439a9e1fc82f153f018c" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240501 07:59:20.669346 6380 leader_election.cc:334] T 72aa4dc7664b48b7a73d34feba7d44d3 P 8315d22c56f7439a9e1fc82f153f018c [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111)
W20240501 07:59:20.720016 6158 catalog_manager.cc:4573] TS 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 144ff11d2a504f7ab7a230e6822d24ca: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111)
W20240501 07:59:20.727272 8728 consensus_peers.cc:485] T c5a876dfb598428ab350593129505504 P 74077b61d6f54342bf9736b4a9f2eade -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:20.732860 13961 raft_consensus.cc:482] T 13daf8496cd64d7d959ce5ad666b9be8 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a505e7cb133c4392bfda8a4a20874ae9)
I20240501 07:59:20.733013 13961 raft_consensus.cc:504] T 13daf8496cd64d7d959ce5ad666b9be8 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } }
I20240501 07:59:20.733538 13961 leader_election.cc:288] T 13daf8496cd64d7d959ce5ad666b9be8 P 8315d22c56f7439a9e1fc82f153f018c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149), a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607)
W20240501 07:59:20.734668 6380 leader_election.cc:334] T 13daf8496cd64d7d959ce5ad666b9be8 P 8315d22c56f7439a9e1fc82f153f018c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111)
W20240501 07:59:20.734776 6380 leader_election.cc:334] T 13daf8496cd64d7d959ce5ad666b9be8 P 8315d22c56f7439a9e1fc82f153f018c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111)
I20240501 07:59:20.734829 6380 leader_election.cc:302] T 13daf8496cd64d7d959ce5ad666b9be8 P 8315d22c56f7439a9e1fc82f153f018c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8315d22c56f7439a9e1fc82f153f018c; no voters: 83f20e84af4c40d09a4804b6b26e697f, a505e7cb133c4392bfda8a4a20874ae9
I20240501 07:59:20.735008 13961 raft_consensus.cc:2726] T 13daf8496cd64d7d959ce5ad666b9be8 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 07:59:20.743680 6575 raft_consensus.cc:1261] T c5a876dfb598428ab350593129505504 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 2 FOLLOWER]: Refusing update from remote peer 74077b61d6f54342bf9736b4a9f2eade: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 07:59:20.744127 14049 consensus_queue.cc:1035] T c5a876dfb598428ab350593129505504 P 74077b61d6f54342bf9736b4a9f2eade [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:59:20.760857 11107 raft_consensus.cc:482] T 12ce6a21d06c4ba889542758be9dc62d P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a505e7cb133c4392bfda8a4a20874ae9)
I20240501 07:59:20.761042 11107 raft_consensus.cc:504] T 12ce6a21d06c4ba889542758be9dc62d P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { replace: true } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: false } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { promote: true } }
I20240501 07:59:20.761582 11107 leader_election.cc:288] T 12ce6a21d06c4ba889542758be9dc62d P 3b164cbbc00f4e8eb3254ddd349a3ff9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607), 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134:40141)
I20240501 07:59:20.762207 8924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12ce6a21d06c4ba889542758be9dc62d" candidate_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "13d05ebdca8c4f489c597ce550dd773d" is_pre_election: true
W20240501 07:59:20.762562 8859 consensus_peers.cc:485] T 8a39f168c8f846cd9b8c4e2f2cf62fdd P 13d05ebdca8c4f489c597ce550dd773d -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 07:59:20.763247 6511 leader_election.cc:334] T 12ce6a21d06c4ba889542758be9dc62d P 3b164cbbc00f4e8eb3254ddd349a3ff9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111)
I20240501 07:59:20.763332 6511 leader_election.cc:302] T 12ce6a21d06c4ba889542758be9dc62d P 3b164cbbc00f4e8eb3254ddd349a3ff9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b164cbbc00f4e8eb3254ddd349a3ff9; no voters: 13d05ebdca8c4f489c597ce550dd773d, a505e7cb133c4392bfda8a4a20874ae9
I20240501 07:59:20.763630 11107 raft_consensus.cc:2726] T 12ce6a21d06c4ba889542758be9dc62d P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 07:59:20.768561 6642 consensus_peers.cc:462] T a29720541b7b4a79aaa810592ef7c4b1 P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111)
I20240501 07:59:20.770388 13418 raft_consensus.cc:482] T 12ce6a21d06c4ba889542758be9dc62d P 13d05ebdca8c4f489c597ce550dd773d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a505e7cb133c4392bfda8a4a20874ae9)
I20240501 07:59:20.770512 13418 raft_consensus.cc:504] T 12ce6a21d06c4ba889542758be9dc62d P 13d05ebdca8c4f489c597ce550dd773d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { replace: true } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: false } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { promote: true } }
I20240501 07:59:20.770917 13418 leader_election.cc:288] T 12ce6a21d06c4ba889542758be9dc62d P 13d05ebdca8c4f489c597ce550dd773d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607), 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131:34471)
I20240501 07:59:20.771407 6575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12ce6a21d06c4ba889542758be9dc62d" candidate_uuid: "13d05ebdca8c4f489c597ce550dd773d" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" is_pre_election: true
I20240501 07:59:20.771601 6575 raft_consensus.cc:2454] T 12ce6a21d06c4ba889542758be9dc62d P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 13d05ebdca8c4f489c597ce550dd773d in term 1.
I20240501 07:59:20.772104 8861 leader_election.cc:302] T 12ce6a21d06c4ba889542758be9dc62d P 13d05ebdca8c4f489c597ce550dd773d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13d05ebdca8c4f489c597ce550dd773d, 3b164cbbc00f4e8eb3254ddd349a3ff9; no voters:
I20240501 07:59:20.772279 13418 raft_consensus.cc:2781] T 12ce6a21d06c4ba889542758be9dc62d P 13d05ebdca8c4f489c597ce550dd773d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 07:59:20.772334 13418 raft_consensus.cc:482] T 12ce6a21d06c4ba889542758be9dc62d P 13d05ebdca8c4f489c597ce550dd773d [term 1 FOLLOWER]: Starting leader election (detected failure of leader a505e7cb133c4392bfda8a4a20874ae9)
I20240501 07:59:20.772385 13418 raft_consensus.cc:3037] T 12ce6a21d06c4ba889542758be9dc62d P 13d05ebdca8c4f489c597ce550dd773d [term 1 FOLLOWER]: Advancing to term 2
W20240501 07:59:20.772691 8859 leader_election.cc:334] T 12ce6a21d06c4ba889542758be9dc62d P 13d05ebdca8c4f489c597ce550dd773d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111)
I20240501 07:59:20.773330 13418 raft_consensus.cc:504] T 12ce6a21d06c4ba889542758be9dc62d P 13d05ebdca8c4f489c597ce550dd773d [term 2 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { replace: true } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: false } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { promote: true } }
I20240501 07:59:20.773670 13418 leader_election.cc:288] T 12ce6a21d06c4ba889542758be9dc62d P 13d05ebdca8c4f489c597ce550dd773d [CANDIDATE]: Term 2 election: Requested vote from peers a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607), 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131:34471)
I20240501 07:59:20.774466 6575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12ce6a21d06c4ba889542758be9dc62d" candidate_uuid: "13d05ebdca8c4f489c597ce550dd773d" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9"
I20240501 07:59:20.774591 6575 raft_consensus.cc:3037] T 12ce6a21d06c4ba889542758be9dc62d P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Advancing to term 2
W20240501 07:59:20.774770 8859 leader_election.cc:334] T 12ce6a21d06c4ba889542758be9dc62d P 13d05ebdca8c4f489c597ce550dd773d [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111)
I20240501 07:59:20.775477 6575 raft_consensus.cc:2454] T 12ce6a21d06c4ba889542758be9dc62d P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13d05ebdca8c4f489c597ce550dd773d in term 2.
I20240501 07:59:20.775766 8861 leader_election.cc:302] T 12ce6a21d06c4ba889542758be9dc62d P 13d05ebdca8c4f489c597ce550dd773d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 13d05ebdca8c4f489c597ce550dd773d, 3b164cbbc00f4e8eb3254ddd349a3ff9; no voters: a505e7cb133c4392bfda8a4a20874ae9
I20240501 07:59:20.775929 13418 raft_consensus.cc:2781] T 12ce6a21d06c4ba889542758be9dc62d P 13d05ebdca8c4f489c597ce550dd773d [term 2 FOLLOWER]: Leader election won for term 2
I20240501 07:59:20.776017 13418 raft_consensus.cc:686] T 12ce6a21d06c4ba889542758be9dc62d P 13d05ebdca8c4f489c597ce550dd773d [term 2 LEADER]: Becoming Leader. State: Replica: 13d05ebdca8c4f489c597ce550dd773d, State: Running, Role: LEADER
I20240501 07:59:20.776160 13418 consensus_queue.cc:237] T 12ce6a21d06c4ba889542758be9dc62d P 13d05ebdca8c4f489c597ce550dd773d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { replace: true } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: false } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { promote: true } }
I20240501 07:59:20.778736 6165 catalog_manager.cc:5484] T 12ce6a21d06c4ba889542758be9dc62d P 13d05ebdca8c4f489c597ce550dd773d reported cstate change: term changed from 1 to 2, leader changed from a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135) to 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134). New cstate: current_term: 2 leader_uuid: "13d05ebdca8c4f489c597ce550dd773d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240501 07:59:20.786187 8859 consensus_peers.cc:485] T b4a2db0198774bb2b4f3df769260e32f P 13d05ebdca8c4f489c597ce550dd773d -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:20.817286 13040 consensus_queue.cc:1035] T 8a39f168c8f846cd9b8c4e2f2cf62fdd P 13d05ebdca8c4f489c597ce550dd773d [LEADER]: Connected to new peer: Peer: permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:20.831732 13040 consensus_queue.cc:1035] T b4a2db0198774bb2b4f3df769260e32f P 13d05ebdca8c4f489c597ce550dd773d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:20.843915 13040 raft_consensus.cc:482] T 8f36eefcda3f475d9571e92fdeb52eb4 P 13d05ebdca8c4f489c597ce550dd773d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a505e7cb133c4392bfda8a4a20874ae9)
I20240501 07:59:20.844048 13040 raft_consensus.cc:504] T 8f36eefcda3f475d9571e92fdeb52eb4 P 13d05ebdca8c4f489c597ce550dd773d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { replace: true } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: true } }
I20240501 07:59:20.844502 13040 leader_election.cc:288] T 8f36eefcda3f475d9571e92fdeb52eb4 P 13d05ebdca8c4f489c597ce550dd773d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607), 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130:32941)
I20240501 07:59:20.845160 6444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f36eefcda3f475d9571e92fdeb52eb4" candidate_uuid: "13d05ebdca8c4f489c597ce550dd773d" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "8315d22c56f7439a9e1fc82f153f018c" is_pre_election: true
I20240501 07:59:20.845325 6444 raft_consensus.cc:2454] T 8f36eefcda3f475d9571e92fdeb52eb4 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 13d05ebdca8c4f489c597ce550dd773d in term 1.
W20240501 07:59:20.845604 8859 leader_election.cc:334] T 8f36eefcda3f475d9571e92fdeb52eb4 P 13d05ebdca8c4f489c597ce550dd773d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111)
I20240501 07:59:20.845691 8860 leader_election.cc:302] T 8f36eefcda3f475d9571e92fdeb52eb4 P 13d05ebdca8c4f489c597ce550dd773d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 13d05ebdca8c4f489c597ce550dd773d, 8315d22c56f7439a9e1fc82f153f018c; no voters: a505e7cb133c4392bfda8a4a20874ae9
I20240501 07:59:20.845875 13351 raft_consensus.cc:2781] T 8f36eefcda3f475d9571e92fdeb52eb4 P 13d05ebdca8c4f489c597ce550dd773d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 07:59:20.845924 13351 raft_consensus.cc:482] T 8f36eefcda3f475d9571e92fdeb52eb4 P 13d05ebdca8c4f489c597ce550dd773d [term 1 FOLLOWER]: Starting leader election (detected failure of leader a505e7cb133c4392bfda8a4a20874ae9)
I20240501 07:59:20.845976 13351 raft_consensus.cc:3037] T 8f36eefcda3f475d9571e92fdeb52eb4 P 13d05ebdca8c4f489c597ce550dd773d [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:20.846958 13351 raft_consensus.cc:504] T 8f36eefcda3f475d9571e92fdeb52eb4 P 13d05ebdca8c4f489c597ce550dd773d [term 2 FOLLOWER]: Starting leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { replace: true } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: true } }
I20240501 07:59:20.847255 13351 leader_election.cc:288] T 8f36eefcda3f475d9571e92fdeb52eb4 P 13d05ebdca8c4f489c597ce550dd773d [CANDIDATE]: Term 2 election: Requested vote from peers a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607), 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130:32941)
I20240501 07:59:20.848104 6444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f36eefcda3f475d9571e92fdeb52eb4" candidate_uuid: "13d05ebdca8c4f489c597ce550dd773d" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "8315d22c56f7439a9e1fc82f153f018c"
I20240501 07:59:20.848222 6444 raft_consensus.cc:3037] T 8f36eefcda3f475d9571e92fdeb52eb4 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Advancing to term 2
I20240501 07:59:20.849462 6444 raft_consensus.cc:2454] T 8f36eefcda3f475d9571e92fdeb52eb4 P 8315d22c56f7439a9e1fc82f153f018c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13d05ebdca8c4f489c597ce550dd773d in term 2.
W20240501 07:59:20.849694 8859 leader_election.cc:334] T 8f36eefcda3f475d9571e92fdeb52eb4 P 13d05ebdca8c4f489c597ce550dd773d [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111)
I20240501 07:59:20.851281 8860 leader_election.cc:302] T 8f36eefcda3f475d9571e92fdeb52eb4 P 13d05ebdca8c4f489c597ce550dd773d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 13d05ebdca8c4f489c597ce550dd773d, 8315d22c56f7439a9e1fc82f153f018c; no voters: a505e7cb133c4392bfda8a4a20874ae9
I20240501 07:59:20.852209 13351 raft_consensus.cc:2781] T 8f36eefcda3f475d9571e92fdeb52eb4 P 13d05ebdca8c4f489c597ce550dd773d [term 2 FOLLOWER]: Leader election won for term 2
I20240501 07:59:20.852319 13351 raft_consensus.cc:686] T 8f36eefcda3f475d9571e92fdeb52eb4 P 13d05ebdca8c4f489c597ce550dd773d [term 2 LEADER]: Becoming Leader. State: Replica: 13d05ebdca8c4f489c597ce550dd773d, State: Running, Role: LEADER
I20240501 07:59:20.852458 13351 consensus_queue.cc:237] T 8f36eefcda3f475d9571e92fdeb52eb4 P 13d05ebdca8c4f489c597ce550dd773d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { replace: true } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: true } }
I20240501 07:59:20.855435 6169 catalog_manager.cc:5484] T 8f36eefcda3f475d9571e92fdeb52eb4 P 13d05ebdca8c4f489c597ce550dd773d reported cstate change: term changed from 1 to 2, leader changed from a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135) to 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134). New cstate: current_term: 2 leader_uuid: "13d05ebdca8c4f489c597ce550dd773d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240501 07:59:20.935458 6249 consensus_peers.cc:485] T b7d57b7fd5234f50afbc859bc997f3e1 P 8b8b3ad3aa8f4053b2167dd550f2086f -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 07:59:20.972142 8859 consensus_peers.cc:485] T f788de4df5454d00ac9bb883f0abc9bd P 13d05ebdca8c4f489c597ce550dd773d -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:20.972828 8793 raft_consensus.cc:1261] T c16b7962c18d4e02b850c7c7c678eae4 P 74077b61d6f54342bf9736b4a9f2eade [term 2 FOLLOWER]: Refusing update from remote peer 13d05ebdca8c4f489c597ce550dd773d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240501 07:59:20.973271 13351 consensus_queue.cc:1035] T c16b7962c18d4e02b850c7c7c678eae4 P 13d05ebdca8c4f489c597ce550dd773d [LEADER]: Connected to new peer: Peer: permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20240501 07:59:20.978446 8859 consensus_peers.cc:485] T c16b7962c18d4e02b850c7c7c678eae4 P 13d05ebdca8c4f489c597ce550dd773d -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:20.978425 6311 raft_consensus.cc:3037] T c16b7962c18d4e02b850c7c7c678eae4 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 1 LEARNER]: Advancing to term 2
I20240501 07:59:20.979693 6311 raft_consensus.cc:1261] T c16b7962c18d4e02b850c7c7c678eae4 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 2 LEARNER]: Refusing update from remote peer 13d05ebdca8c4f489c597ce550dd773d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240501 07:59:20.980245 14130 consensus_queue.cc:1035] T c16b7962c18d4e02b850c7c7c678eae4 P 13d05ebdca8c4f489c597ce550dd773d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 07:59:20.982419 14130 raft_consensus.cc:1050] T c16b7962c18d4e02b850c7c7c678eae4 P 13d05ebdca8c4f489c597ce550dd773d: attempting to promote NON_VOTER 8b8b3ad3aa8f4053b2167dd550f2086f to VOTER
I20240501 07:59:20.982892 14130 consensus_queue.cc:237] T c16b7962c18d4e02b850c7c7c678eae4 P 13d05ebdca8c4f489c597ce550dd773d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 2, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { promote: false } }
I20240501 07:59:20.984333 6311 raft_consensus.cc:1261] T c16b7962c18d4e02b850c7c7c678eae4 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 2 LEARNER]: Refusing update from remote peer 13d05ebdca8c4f489c597ce550dd773d: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 07:59:20.984504 8794 raft_consensus.cc:1261] T c16b7962c18d4e02b850c7c7c678eae4 P 74077b61d6f54342bf9736b4a9f2eade [term 2 FOLLOWER]: Refusing update from remote peer 13d05ebdca8c4f489c597ce550dd773d: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 07:59:20.985042 13418 consensus_queue.cc:1035] T c16b7962c18d4e02b850c7c7c678eae4 P 13d05ebdca8c4f489c597ce550dd773d [LEADER]: Connected to new peer: Peer: permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:20.985428 13418 consensus_queue.cc:1035] T c16b7962c18d4e02b850c7c7c678eae4 P 13d05ebdca8c4f489c597ce550dd773d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 07:59:20.986138 8859 consensus_peers.cc:485] T c16b7962c18d4e02b850c7c7c678eae4 P 13d05ebdca8c4f489c597ce550dd773d -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:20.990612 13418 raft_consensus.cc:2932] T c16b7962c18d4e02b850c7c7c678eae4 P 13d05ebdca8c4f489c597ce550dd773d [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 2 to 4, 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { promote: false } } }
I20240501 07:59:20.991149 6311 raft_consensus.cc:2932] T c16b7962c18d4e02b850c7c7c678eae4 P 8b8b3ad3aa8f4053b2167dd550f2086f [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { promote: false } } }
I20240501 07:59:20.992983 8794 raft_consensus.cc:2932] T c16b7962c18d4e02b850c7c7c678eae4 P 74077b61d6f54342bf9736b4a9f2eade [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { promote: false } } }
I20240501 07:59:20.993690 6165 catalog_manager.cc:5484] T c16b7962c18d4e02b850c7c7c678eae4 P 13d05ebdca8c4f489c597ce550dd773d reported cstate change: config changed from index 2 to 4, 8b8b3ad3aa8f4053b2167dd550f2086f (127.0.99.129) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "13d05ebdca8c4f489c597ce550dd773d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 07:59:20.998512 14049 raft_consensus.cc:482] T 7dbbfc5d5848486f983cffdd2f36cc91 P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:20.998623 14049 raft_consensus.cc:504] T 7dbbfc5d5848486f983cffdd2f36cc91 P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } }
I20240501 07:59:20.998986 14049 leader_election.cc:288] T 7dbbfc5d5848486f983cffdd2f36cc91 P 74077b61d6f54342bf9736b4a9f2eade [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149), a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607)
W20240501 07:59:20.999996 8728 leader_election.cc:334] T 7dbbfc5d5848486f983cffdd2f36cc91 P 74077b61d6f54342bf9736b4a9f2eade [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111)
W20240501 07:59:21.000110 8728 leader_election.cc:334] T 7dbbfc5d5848486f983cffdd2f36cc91 P 74077b61d6f54342bf9736b4a9f2eade [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111)
I20240501 07:59:21.000167 8728 leader_election.cc:302] T 7dbbfc5d5848486f983cffdd2f36cc91 P 74077b61d6f54342bf9736b4a9f2eade [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74077b61d6f54342bf9736b4a9f2eade; no voters: 83f20e84af4c40d09a4804b6b26e697f, a505e7cb133c4392bfda8a4a20874ae9
I20240501 07:59:21.000320 14049 raft_consensus.cc:2726] T 7dbbfc5d5848486f983cffdd2f36cc91 P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 07:59:21.040095 6511 consensus_peers.cc:485] T af718dc43add4edfac703013655699f4 P 3b164cbbc00f4e8eb3254ddd349a3ff9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:21.044867 8924 raft_consensus.cc:1261] T f88fe033bce14bf0b1b63a52a7ba137e P 13d05ebdca8c4f489c597ce550dd773d [term 2 FOLLOWER]: Refusing update from remote peer 8315d22c56f7439a9e1fc82f153f018c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 07:59:21.045328 13961 consensus_queue.cc:1035] T f88fe033bce14bf0b1b63a52a7ba137e P 8315d22c56f7439a9e1fc82f153f018c [LEADER]: Connected to new peer: Peer: permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:59:21.045574 6575 raft_consensus.cc:3037] T 2560b1ec81ac4357b32c7a8ba9d72216 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 LEARNER]: Advancing to term 2
I20240501 07:59:21.046761 6575 raft_consensus.cc:1261] T 2560b1ec81ac4357b32c7a8ba9d72216 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 2 LEARNER]: Refusing update from remote peer 1f9cb72be98941b5a8e504cbe34132b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240501 07:59:21.047240 14067 consensus_queue.cc:1035] T 2560b1ec81ac4357b32c7a8ba9d72216 P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20240501 07:59:21.049436 6380 consensus_peers.cc:485] T f88fe033bce14bf0b1b63a52a7ba137e P 8315d22c56f7439a9e1fc82f153f018c -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:21.051352 14067 raft_consensus.cc:1050] T 2560b1ec81ac4357b32c7a8ba9d72216 P 1f9cb72be98941b5a8e504cbe34132b9: attempting to promote NON_VOTER 3b164cbbc00f4e8eb3254ddd349a3ff9 to VOTER
W20240501 07:59:21.051443 14067 raft_consensus.cc:1053] T 2560b1ec81ac4357b32c7a8ba9d72216 P 1f9cb72be98941b5a8e504cbe34132b9: Unable to promote non-voter 3b164cbbc00f4e8eb3254ddd349a3ff9: Illegal state: Leader has not yet committed an operation in its own term
I20240501 07:59:21.055140 8794 raft_consensus.cc:1261] T b7d57b7fd5234f50afbc859bc997f3e1 P 74077b61d6f54342bf9736b4a9f2eade [term 3 FOLLOWER]: Refusing update from remote peer 8b8b3ad3aa8f4053b2167dd550f2086f: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 3 index: 11. (index mismatch)
I20240501 07:59:21.055557 14068 consensus_queue.cc:1035] T b7d57b7fd5234f50afbc859bc997f3e1 P 8b8b3ad3aa8f4053b2167dd550f2086f [LEADER]: Connected to new peer: Peer: permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 07:59:21.063889 13351 consensus_queue.cc:1035] T f788de4df5454d00ac9bb883f0abc9bd P 13d05ebdca8c4f489c597ce550dd773d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 07:59:21.074736 6172 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55950:
name: "rebalancer_extra_table_34"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 07:59:21.081770 6424 tablet_service.cc:1467] Processing CreateTablet for tablet d726533fb72d4827991e12034b42634e (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=2a4130566cd944b2a8a081830116ea7c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:21.081934 6556 tablet_service.cc:1467] Processing CreateTablet for tablet e81b23661a1d475887c63f79d8f90ab6 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=2a4130566cd944b2a8a081830116ea7c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:21.082106 6424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d726533fb72d4827991e12034b42634e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:21.082182 6556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e81b23661a1d475887c63f79d8f90ab6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:21.082124 8757 tablet_service.cc:1467] Processing CreateTablet for tablet d726533fb72d4827991e12034b42634e (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=2a4130566cd944b2a8a081830116ea7c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:21.082370 8757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d726533fb72d4827991e12034b42634e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:21.083753 6425 tablet_service.cc:1467] Processing CreateTablet for tablet 4664b04b23294ad4b0a0e5c87fdca2ad (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=2a4130566cd944b2a8a081830116ea7c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:21.083993 6425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4664b04b23294ad4b0a0e5c87fdca2ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:21.086331 8905 tablet_service.cc:1467] Processing CreateTablet for tablet 4664b04b23294ad4b0a0e5c87fdca2ad (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=2a4130566cd944b2a8a081830116ea7c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 07:59:21.086658 8905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4664b04b23294ad4b0a0e5c87fdca2ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 07:59:21.086663 14142 tablet_bootstrap.cc:492] T d726533fb72d4827991e12034b42634e P 74077b61d6f54342bf9736b4a9f2eade: Bootstrap starting.
I20240501 07:59:21.087188 14141 tablet_bootstrap.cc:492] T d726533fb72d4827991e12034b42634e P 8315d22c56f7439a9e1fc82f153f018c: Bootstrap starting.
I20240501 07:59:21.087852 14142 tablet_bootstrap.cc:654] T d726533fb72d4827991e12034b42634e P 74077b61d6f54342bf9736b4a9f2eade: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:21.088204 14141 tablet_bootstrap.cc:654] T d726533fb72d4827991e12034b42634e P 8315d22c56f7439a9e1fc82f153f018c: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:21.089222 14140 tablet_bootstrap.cc:492] T e81b23661a1d475887c63f79d8f90ab6 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Bootstrap starting.
I20240501 07:59:21.090111 14141 tablet_bootstrap.cc:492] T d726533fb72d4827991e12034b42634e P 8315d22c56f7439a9e1fc82f153f018c: No bootstrap required, opened a new log
I20240501 07:59:21.090241 14141 ts_tablet_manager.cc:1367] T d726533fb72d4827991e12034b42634e P 8315d22c56f7439a9e1fc82f153f018c: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 07:59:21.090334 14140 tablet_bootstrap.cc:654] T e81b23661a1d475887c63f79d8f90ab6 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:21.090780 14141 raft_consensus.cc:348] T d726533fb72d4827991e12034b42634e P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } }
I20240501 07:59:21.091001 14141 raft_consensus.cc:374] T d726533fb72d4827991e12034b42634e P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:21.091092 14141 raft_consensus.cc:729] T d726533fb72d4827991e12034b42634e P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8315d22c56f7439a9e1fc82f153f018c, State: Initialized, Role: FOLLOWER
I20240501 07:59:21.091306 14141 consensus_queue.cc:260] T d726533fb72d4827991e12034b42634e P 8315d22c56f7439a9e1fc82f153f018c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } }
I20240501 07:59:21.091452 14140 tablet_bootstrap.cc:492] T e81b23661a1d475887c63f79d8f90ab6 P 3b164cbbc00f4e8eb3254ddd349a3ff9: No bootstrap required, opened a new log
I20240501 07:59:21.091456 14142 tablet_bootstrap.cc:492] T d726533fb72d4827991e12034b42634e P 74077b61d6f54342bf9736b4a9f2eade: No bootstrap required, opened a new log
I20240501 07:59:21.091547 14142 ts_tablet_manager.cc:1367] T d726533fb72d4827991e12034b42634e P 74077b61d6f54342bf9736b4a9f2eade: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 07:59:21.091596 14140 ts_tablet_manager.cc:1367] T e81b23661a1d475887c63f79d8f90ab6 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 07:59:21.092005 14140 raft_consensus.cc:348] T e81b23661a1d475887c63f79d8f90ab6 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } }
I20240501 07:59:21.092005 14142 raft_consensus.cc:348] T d726533fb72d4827991e12034b42634e P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } }
I20240501 07:59:21.092108 14140 raft_consensus.cc:374] T e81b23661a1d475887c63f79d8f90ab6 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:21.092118 14142 raft_consensus.cc:374] T d726533fb72d4827991e12034b42634e P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:21.092145 14140 raft_consensus.cc:729] T e81b23661a1d475887c63f79d8f90ab6 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b164cbbc00f4e8eb3254ddd349a3ff9, State: Initialized, Role: FOLLOWER
I20240501 07:59:21.092152 14142 raft_consensus.cc:729] T d726533fb72d4827991e12034b42634e P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74077b61d6f54342bf9736b4a9f2eade, State: Initialized, Role: FOLLOWER
I20240501 07:59:21.092275 14142 consensus_queue.cc:260] T d726533fb72d4827991e12034b42634e P 74077b61d6f54342bf9736b4a9f2eade [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } }
I20240501 07:59:21.092300 14140 consensus_queue.cc:260] T e81b23661a1d475887c63f79d8f90ab6 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } }
I20240501 07:59:21.092568 14142 ts_tablet_manager.cc:1398] T d726533fb72d4827991e12034b42634e P 74077b61d6f54342bf9736b4a9f2eade: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 07:59:21.092739 14141 ts_tablet_manager.cc:1398] T d726533fb72d4827991e12034b42634e P 8315d22c56f7439a9e1fc82f153f018c: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:59:21.092885 14141 tablet_bootstrap.cc:492] T 4664b04b23294ad4b0a0e5c87fdca2ad P 8315d22c56f7439a9e1fc82f153f018c: Bootstrap starting.
I20240501 07:59:21.093446 14140 ts_tablet_manager.cc:1398] T e81b23661a1d475887c63f79d8f90ab6 P 3b164cbbc00f4e8eb3254ddd349a3ff9: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:59:21.093910 14141 tablet_bootstrap.cc:654] T 4664b04b23294ad4b0a0e5c87fdca2ad P 8315d22c56f7439a9e1fc82f153f018c: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:21.094794 14145 tablet_bootstrap.cc:492] T 4664b04b23294ad4b0a0e5c87fdca2ad P 13d05ebdca8c4f489c597ce550dd773d: Bootstrap starting.
I20240501 07:59:21.095511 8794 raft_consensus.cc:1261] T 2560b1ec81ac4357b32c7a8ba9d72216 P 74077b61d6f54342bf9736b4a9f2eade [term 2 FOLLOWER]: Refusing update from remote peer 1f9cb72be98941b5a8e504cbe34132b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240501 07:59:21.095813 14145 tablet_bootstrap.cc:654] T 4664b04b23294ad4b0a0e5c87fdca2ad P 13d05ebdca8c4f489c597ce550dd773d: Neither blocks nor log segments found. Creating new log.
I20240501 07:59:21.095893 14067 consensus_queue.cc:1035] T 2560b1ec81ac4357b32c7a8ba9d72216 P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 07:59:21.097972 14145 tablet_bootstrap.cc:492] T 4664b04b23294ad4b0a0e5c87fdca2ad P 13d05ebdca8c4f489c597ce550dd773d: No bootstrap required, opened a new log
I20240501 07:59:21.098248 14145 ts_tablet_manager.cc:1367] T 4664b04b23294ad4b0a0e5c87fdca2ad P 13d05ebdca8c4f489c597ce550dd773d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 07:59:21.099232 14145 raft_consensus.cc:348] T 4664b04b23294ad4b0a0e5c87fdca2ad P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } }
I20240501 07:59:21.099334 14145 raft_consensus.cc:374] T 4664b04b23294ad4b0a0e5c87fdca2ad P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:21.099367 14145 raft_consensus.cc:729] T 4664b04b23294ad4b0a0e5c87fdca2ad P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13d05ebdca8c4f489c597ce550dd773d, State: Initialized, Role: FOLLOWER
I20240501 07:59:21.099498 14145 consensus_queue.cc:260] T 4664b04b23294ad4b0a0e5c87fdca2ad P 13d05ebdca8c4f489c597ce550dd773d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } }
I20240501 07:59:21.099881 14145 ts_tablet_manager.cc:1398] T 4664b04b23294ad4b0a0e5c87fdca2ad P 13d05ebdca8c4f489c597ce550dd773d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 07:59:21.100446 14141 tablet_bootstrap.cc:492] T 4664b04b23294ad4b0a0e5c87fdca2ad P 8315d22c56f7439a9e1fc82f153f018c: No bootstrap required, opened a new log
I20240501 07:59:21.100540 14141 ts_tablet_manager.cc:1367] T 4664b04b23294ad4b0a0e5c87fdca2ad P 8315d22c56f7439a9e1fc82f153f018c: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240501 07:59:21.101078 14141 raft_consensus.cc:348] T 4664b04b23294ad4b0a0e5c87fdca2ad P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } }
I20240501 07:59:21.101202 14141 raft_consensus.cc:374] T 4664b04b23294ad4b0a0e5c87fdca2ad P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 07:59:21.101235 14141 raft_consensus.cc:729] T 4664b04b23294ad4b0a0e5c87fdca2ad P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8315d22c56f7439a9e1fc82f153f018c, State: Initialized, Role: FOLLOWER
I20240501 07:59:21.101348 14141 consensus_queue.cc:260] T 4664b04b23294ad4b0a0e5c87fdca2ad P 8315d22c56f7439a9e1fc82f153f018c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } }
I20240501 07:59:21.103461 14141 ts_tablet_manager.cc:1398] T 4664b04b23294ad4b0a0e5c87fdca2ad P 8315d22c56f7439a9e1fc82f153f018c: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
W20240501 07:59:21.103904 6642 consensus_peers.cc:485] T 2560b1ec81ac4357b32c7a8ba9d72216 P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:21.104212 13495 raft_consensus.cc:1050] T 2560b1ec81ac4357b32c7a8ba9d72216 P 1f9cb72be98941b5a8e504cbe34132b9: attempting to promote NON_VOTER 3b164cbbc00f4e8eb3254ddd349a3ff9 to VOTER
I20240501 07:59:21.104604 13495 consensus_queue.cc:237] T 2560b1ec81ac4357b32c7a8ba9d72216 P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 2, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: false } }
I20240501 07:59:21.106572 6575 raft_consensus.cc:1261] T 2560b1ec81ac4357b32c7a8ba9d72216 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 2 LEARNER]: Refusing update from remote peer 1f9cb72be98941b5a8e504cbe34132b9: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 07:59:21.106838 8794 raft_consensus.cc:1261] T 2560b1ec81ac4357b32c7a8ba9d72216 P 74077b61d6f54342bf9736b4a9f2eade [term 2 FOLLOWER]: Refusing update from remote peer 1f9cb72be98941b5a8e504cbe34132b9: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
W20240501 07:59:21.107452 6642 consensus_peers.cc:485] T 2560b1ec81ac4357b32c7a8ba9d72216 P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:21.107625 13495 consensus_queue.cc:1035] T 2560b1ec81ac4357b32c7a8ba9d72216 P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:21.107683 14135 raft_consensus.cc:482] T d726533fb72d4827991e12034b42634e P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:21.107867 14067 consensus_queue.cc:1035] T 2560b1ec81ac4357b32c7a8ba9d72216 P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:21.108091 14135 raft_consensus.cc:504] T d726533fb72d4827991e12034b42634e P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } }
I20240501 07:59:21.108599 14135 leader_election.cc:288] T d726533fb72d4827991e12034b42634e P 8315d22c56f7439a9e1fc82f153f018c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133:39859), 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149)
I20240501 07:59:21.108843 8794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d726533fb72d4827991e12034b42634e" candidate_uuid: "8315d22c56f7439a9e1fc82f153f018c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74077b61d6f54342bf9736b4a9f2eade" is_pre_election: true
I20240501 07:59:21.109031 8794 raft_consensus.cc:2454] T d726533fb72d4827991e12034b42634e P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8315d22c56f7439a9e1fc82f153f018c in term 0.
I20240501 07:59:21.109306 6379 leader_election.cc:302] T d726533fb72d4827991e12034b42634e P 8315d22c56f7439a9e1fc82f153f018c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74077b61d6f54342bf9736b4a9f2eade, 8315d22c56f7439a9e1fc82f153f018c; no voters:
I20240501 07:59:21.109460 14135 raft_consensus.cc:2781] T d726533fb72d4827991e12034b42634e P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:21.109519 14135 raft_consensus.cc:482] T d726533fb72d4827991e12034b42634e P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:21.109562 14135 raft_consensus.cc:3037] T d726533fb72d4827991e12034b42634e P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:21.110741 14135 raft_consensus.cc:504] T d726533fb72d4827991e12034b42634e P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } }
W20240501 07:59:21.110893 6380 leader_election.cc:334] T d726533fb72d4827991e12034b42634e P 8315d22c56f7439a9e1fc82f153f018c [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111)
I20240501 07:59:21.111459 8924 raft_consensus.cc:1261] T 72aa4dc7664b48b7a73d34feba7d44d3 P 13d05ebdca8c4f489c597ce550dd773d [term 3 FOLLOWER]: Refusing update from remote peer 8315d22c56f7439a9e1fc82f153f018c: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20240501 07:59:21.111439 8794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d726533fb72d4827991e12034b42634e" candidate_uuid: "8315d22c56f7439a9e1fc82f153f018c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74077b61d6f54342bf9736b4a9f2eade"
I20240501 07:59:21.111564 8794 raft_consensus.cc:3037] T d726533fb72d4827991e12034b42634e P 74077b61d6f54342bf9736b4a9f2eade [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:21.111881 13961 consensus_queue.cc:1035] T 72aa4dc7664b48b7a73d34feba7d44d3 P 8315d22c56f7439a9e1fc82f153f018c [LEADER]: Connected to new peer: Peer: permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240501 07:59:21.112612 6380 leader_election.cc:334] T d726533fb72d4827991e12034b42634e P 8315d22c56f7439a9e1fc82f153f018c [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111)
I20240501 07:59:21.112725 14135 leader_election.cc:288] T d726533fb72d4827991e12034b42634e P 8315d22c56f7439a9e1fc82f153f018c [CANDIDATE]: Term 1 election: Requested vote from peers 74077b61d6f54342bf9736b4a9f2eade (127.0.99.133:39859), 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149)
I20240501 07:59:21.112762 8794 raft_consensus.cc:2454] T d726533fb72d4827991e12034b42634e P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8315d22c56f7439a9e1fc82f153f018c in term 1.
I20240501 07:59:21.113211 6379 leader_election.cc:302] T d726533fb72d4827991e12034b42634e P 8315d22c56f7439a9e1fc82f153f018c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 74077b61d6f54342bf9736b4a9f2eade, 8315d22c56f7439a9e1fc82f153f018c; no voters: 83f20e84af4c40d09a4804b6b26e697f
I20240501 07:59:21.113533 14135 raft_consensus.cc:2781] T d726533fb72d4827991e12034b42634e P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:21.113690 14135 raft_consensus.cc:686] T d726533fb72d4827991e12034b42634e P 8315d22c56f7439a9e1fc82f153f018c [term 1 LEADER]: Becoming Leader. State: Replica: 8315d22c56f7439a9e1fc82f153f018c, State: Running, Role: LEADER
I20240501 07:59:21.113866 14135 consensus_queue.cc:237] T d726533fb72d4827991e12034b42634e P 8315d22c56f7439a9e1fc82f153f018c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } }
W20240501 07:59:21.114727 6511 consensus_peers.cc:462] T 0b24f67cf1a64debb23ee6618c820d33 P 3b164cbbc00f4e8eb3254ddd349a3ff9 -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111)
I20240501 07:59:21.115057 14152 raft_consensus.cc:2932] T 2560b1ec81ac4357b32c7a8ba9d72216 P 1f9cb72be98941b5a8e504cbe34132b9 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 2 to 4, 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: false } } }
I20240501 07:59:21.115535 8793 raft_consensus.cc:2932] T 2560b1ec81ac4357b32c7a8ba9d72216 P 74077b61d6f54342bf9736b4a9f2eade [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: false } } }
I20240501 07:59:21.117496 6168 catalog_manager.cc:5484] T d726533fb72d4827991e12034b42634e P 8315d22c56f7439a9e1fc82f153f018c reported cstate change: term changed from 0 to 1, leader changed from <none> to 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130). New cstate: current_term: 1 leader_uuid: "8315d22c56f7439a9e1fc82f153f018c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } health_report { overall_health: UNKNOWN } } }
I20240501 07:59:21.117878 6575 raft_consensus.cc:2932] T 2560b1ec81ac4357b32c7a8ba9d72216 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: false } } }
I20240501 07:59:21.119103 6172 catalog_manager.cc:5484] T 2560b1ec81ac4357b32c7a8ba9d72216 P 74077b61d6f54342bf9736b4a9f2eade reported cstate change: config changed from index 2 to 4, 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "1f9cb72be98941b5a8e504cbe34132b9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { replace: true } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: false } } }
I20240501 07:59:21.119900 14135 raft_consensus.cc:482] T 4664b04b23294ad4b0a0e5c87fdca2ad P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:21.120029 14135 raft_consensus.cc:504] T 4664b04b23294ad4b0a0e5c87fdca2ad P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } }
W20240501 07:59:21.123101 6380 leader_election.cc:334] T 4664b04b23294ad4b0a0e5c87fdca2ad P 8315d22c56f7439a9e1fc82f153f018c [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111)
I20240501 07:59:21.123335 8925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4664b04b23294ad4b0a0e5c87fdca2ad" candidate_uuid: "8315d22c56f7439a9e1fc82f153f018c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13d05ebdca8c4f489c597ce550dd773d" is_pre_election: true
I20240501 07:59:21.123479 8925 raft_consensus.cc:2454] T 4664b04b23294ad4b0a0e5c87fdca2ad P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8315d22c56f7439a9e1fc82f153f018c in term 0.
I20240501 07:59:21.123876 8793 raft_consensus.cc:1261] T af718dc43add4edfac703013655699f4 P 74077b61d6f54342bf9736b4a9f2eade [term 2 FOLLOWER]: Refusing update from remote peer 3b164cbbc00f4e8eb3254ddd349a3ff9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 07:59:21.123957 6381 leader_election.cc:302] T 4664b04b23294ad4b0a0e5c87fdca2ad P 8315d22c56f7439a9e1fc82f153f018c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 13d05ebdca8c4f489c597ce550dd773d, 8315d22c56f7439a9e1fc82f153f018c; no voters: a505e7cb133c4392bfda8a4a20874ae9
I20240501 07:59:21.124118 13961 raft_consensus.cc:2781] T 4664b04b23294ad4b0a0e5c87fdca2ad P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 07:59:21.124173 13961 raft_consensus.cc:482] T 4664b04b23294ad4b0a0e5c87fdca2ad P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:21.124212 13961 raft_consensus.cc:3037] T 4664b04b23294ad4b0a0e5c87fdca2ad P 8315d22c56f7439a9e1fc82f153f018c [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:21.125161 13961 raft_consensus.cc:504] T 4664b04b23294ad4b0a0e5c87fdca2ad P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } }
I20240501 07:59:21.125321 13956 consensus_queue.cc:1035] T af718dc43add4edfac703013655699f4 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 07:59:21.125485 13961 leader_election.cc:288] T 4664b04b23294ad4b0a0e5c87fdca2ad P 8315d22c56f7439a9e1fc82f153f018c [CANDIDATE]: Term 1 election: Requested vote from peers a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607), 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134:40141)
I20240501 07:59:21.125654 14135 leader_election.cc:288] T 4664b04b23294ad4b0a0e5c87fdca2ad P 8315d22c56f7439a9e1fc82f153f018c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607), 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134:40141)
I20240501 07:59:21.125919 8924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4664b04b23294ad4b0a0e5c87fdca2ad" candidate_uuid: "8315d22c56f7439a9e1fc82f153f018c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13d05ebdca8c4f489c597ce550dd773d"
I20240501 07:59:21.126034 8924 raft_consensus.cc:3037] T 4664b04b23294ad4b0a0e5c87fdca2ad P 13d05ebdca8c4f489c597ce550dd773d [term 0 FOLLOWER]: Advancing to term 1
I20240501 07:59:21.127005 8924 raft_consensus.cc:2454] T 4664b04b23294ad4b0a0e5c87fdca2ad P 13d05ebdca8c4f489c597ce550dd773d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8315d22c56f7439a9e1fc82f153f018c in term 1.
I20240501 07:59:21.127908 6381 leader_election.cc:302] T 4664b04b23294ad4b0a0e5c87fdca2ad P 8315d22c56f7439a9e1fc82f153f018c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13d05ebdca8c4f489c597ce550dd773d, 8315d22c56f7439a9e1fc82f153f018c; no voters:
W20240501 07:59:21.128000 6380 leader_election.cc:334] T 4664b04b23294ad4b0a0e5c87fdca2ad P 8315d22c56f7439a9e1fc82f153f018c [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111)
W20240501 07:59:21.128123 6380 consensus_peers.cc:485] T 72aa4dc7664b48b7a73d34feba7d44d3 P 8315d22c56f7439a9e1fc82f153f018c -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:21.128131 13961 raft_consensus.cc:2781] T 4664b04b23294ad4b0a0e5c87fdca2ad P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Leader election won for term 1
I20240501 07:59:21.128252 13961 raft_consensus.cc:686] T 4664b04b23294ad4b0a0e5c87fdca2ad P 8315d22c56f7439a9e1fc82f153f018c [term 1 LEADER]: Becoming Leader. State: Replica: 8315d22c56f7439a9e1fc82f153f018c, State: Running, Role: LEADER
I20240501 07:59:21.128368 13961 consensus_queue.cc:237] T 4664b04b23294ad4b0a0e5c87fdca2ad P 8315d22c56f7439a9e1fc82f153f018c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } }
I20240501 07:59:21.130759 6168 catalog_manager.cc:5484] T 4664b04b23294ad4b0a0e5c87fdca2ad P 8315d22c56f7439a9e1fc82f153f018c reported cstate change: term changed from 0 to 1, leader changed from <none> to 8315d22c56f7439a9e1fc82f153f018c (127.0.99.130). New cstate: current_term: 1 leader_uuid: "8315d22c56f7439a9e1fc82f153f018c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } health_report { overall_health: HEALTHY } } }
I20240501 07:59:21.134779 11107 raft_consensus.cc:482] T e81b23661a1d475887c63f79d8f90ab6 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:21.134874 11107 raft_consensus.cc:504] T e81b23661a1d475887c63f79d8f90ab6 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } }
I20240501 07:59:21.135193 11107 leader_election.cc:288] T e81b23661a1d475887c63f79d8f90ab6 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149), a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607)
W20240501 07:59:21.136443 6511 leader_election.cc:334] T e81b23661a1d475887c63f79d8f90ab6 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111)
W20240501 07:59:21.136536 6511 leader_election.cc:334] T e81b23661a1d475887c63f79d8f90ab6 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111)
I20240501 07:59:21.136571 6511 leader_election.cc:302] T e81b23661a1d475887c63f79d8f90ab6 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b164cbbc00f4e8eb3254ddd349a3ff9; no voters: 83f20e84af4c40d09a4804b6b26e697f, a505e7cb133c4392bfda8a4a20874ae9
I20240501 07:59:21.136677 11107 raft_consensus.cc:2726] T e81b23661a1d475887c63f79d8f90ab6 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 07:59:21.153532 6575 raft_consensus.cc:1261] T 12ce6a21d06c4ba889542758be9dc62d P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 2 FOLLOWER]: Refusing update from remote peer 13d05ebdca8c4f489c597ce550dd773d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 07:59:21.154065 13351 consensus_queue.cc:1035] T 12ce6a21d06c4ba889542758be9dc62d P 13d05ebdca8c4f489c597ce550dd773d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 07:59:21.156368 6707 raft_consensus.cc:3037] T 12ce6a21d06c4ba889542758be9dc62d P 1f9cb72be98941b5a8e504cbe34132b9 [term 1 LEARNER]: Advancing to term 2
I20240501 07:59:21.157614 6707 raft_consensus.cc:1261] T 12ce6a21d06c4ba889542758be9dc62d P 1f9cb72be98941b5a8e504cbe34132b9 [term 2 LEARNER]: Refusing update from remote peer 13d05ebdca8c4f489c597ce550dd773d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 07:59:21.158051 13418 consensus_queue.cc:1035] T 12ce6a21d06c4ba889542758be9dc62d P 13d05ebdca8c4f489c597ce550dd773d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240501 07:59:21.159157 8859 consensus_peers.cc:485] T 12ce6a21d06c4ba889542758be9dc62d P 13d05ebdca8c4f489c597ce550dd773d -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:21.160527 13418 raft_consensus.cc:1050] T 12ce6a21d06c4ba889542758be9dc62d P 13d05ebdca8c4f489c597ce550dd773d: attempting to promote NON_VOTER 1f9cb72be98941b5a8e504cbe34132b9 to VOTER
I20240501 07:59:21.160938 13418 consensus_queue.cc:237] T 12ce6a21d06c4ba889542758be9dc62d P 13d05ebdca8c4f489c597ce550dd773d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { replace: true } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: false } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { promote: false } }
I20240501 07:59:21.162318 6575 raft_consensus.cc:1261] T 12ce6a21d06c4ba889542758be9dc62d P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 2 FOLLOWER]: Refusing update from remote peer 13d05ebdca8c4f489c597ce550dd773d: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 07:59:21.163306 13040 consensus_queue.cc:1035] T 12ce6a21d06c4ba889542758be9dc62d P 13d05ebdca8c4f489c597ce550dd773d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240501 07:59:21.164464 8859 consensus_peers.cc:485] T 12ce6a21d06c4ba889542758be9dc62d P 13d05ebdca8c4f489c597ce550dd773d -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:21.165714 6707 raft_consensus.cc:1261] T 12ce6a21d06c4ba889542758be9dc62d P 1f9cb72be98941b5a8e504cbe34132b9 [term 2 LEARNER]: Refusing update from remote peer 13d05ebdca8c4f489c597ce550dd773d: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 07:59:21.166121 13351 consensus_queue.cc:1035] T 12ce6a21d06c4ba889542758be9dc62d P 13d05ebdca8c4f489c597ce550dd773d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 07:59:21.167657 13418 raft_consensus.cc:2932] T 12ce6a21d06c4ba889542758be9dc62d P 13d05ebdca8c4f489c597ce550dd773d [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { replace: true } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: false } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { promote: false } } }
I20240501 07:59:21.168093 6707 raft_consensus.cc:2932] T 12ce6a21d06c4ba889542758be9dc62d P 1f9cb72be98941b5a8e504cbe34132b9 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { replace: true } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: false } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { promote: false } } }
I20240501 07:59:21.170143 6575 raft_consensus.cc:2932] T 12ce6a21d06c4ba889542758be9dc62d P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { replace: true } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: false } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { promote: false } } }
I20240501 07:59:21.170845 6168 catalog_manager.cc:5484] T 12ce6a21d06c4ba889542758be9dc62d P 1f9cb72be98941b5a8e504cbe34132b9 reported cstate change: config changed from index 5 to 7, 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "13d05ebdca8c4f489c597ce550dd773d" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { replace: true } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: false } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } attrs { promote: false } } }
I20240501 07:59:21.171926 6313 raft_consensus.cc:1261] T e5bd3a8f7a7640c09774f781009eb19c P 8b8b3ad3aa8f4053b2167dd550f2086f [term 2 FOLLOWER]: Refusing update from remote peer 74077b61d6f54342bf9736b4a9f2eade: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 07:59:21.172366 14049 consensus_queue.cc:1035] T e5bd3a8f7a7640c09774f781009eb19c P 74077b61d6f54342bf9736b4a9f2eade [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 07:59:21.178069 8728 consensus_peers.cc:485] T e5bd3a8f7a7640c09774f781009eb19c P 74077b61d6f54342bf9736b4a9f2eade -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 07:59:21.282740 6642 consensus_peers.cc:462] T a29720541b7b4a79aaa810592ef7c4b1 P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111)
W20240501 07:59:21.362497 8859 consensus_peers.cc:485] T 8f36eefcda3f475d9571e92fdeb52eb4 P 13d05ebdca8c4f489c597ce550dd773d -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:21.383100 6444 raft_consensus.cc:1261] T 8f36eefcda3f475d9571e92fdeb52eb4 P 8315d22c56f7439a9e1fc82f153f018c [term 2 FOLLOWER]: Refusing update from remote peer 13d05ebdca8c4f489c597ce550dd773d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240501 07:59:21.383519 13351 consensus_queue.cc:1035] T 8f36eefcda3f475d9571e92fdeb52eb4 P 13d05ebdca8c4f489c597ce550dd773d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 07:59:21.386958 6575 raft_consensus.cc:3037] T 8f36eefcda3f475d9571e92fdeb52eb4 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 LEARNER]: Advancing to term 2
I20240501 07:59:21.388110 6575 raft_consensus.cc:1261] T 8f36eefcda3f475d9571e92fdeb52eb4 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 2 LEARNER]: Refusing update from remote peer 13d05ebdca8c4f489c597ce550dd773d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240501 07:59:21.388579 13418 consensus_queue.cc:1035] T 8f36eefcda3f475d9571e92fdeb52eb4 P 13d05ebdca8c4f489c597ce550dd773d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 07:59:21.390561 13040 raft_consensus.cc:1050] T 8f36eefcda3f475d9571e92fdeb52eb4 P 13d05ebdca8c4f489c597ce550dd773d: attempting to promote NON_VOTER 3b164cbbc00f4e8eb3254ddd349a3ff9 to VOTER
I20240501 07:59:21.390903 13040 consensus_queue.cc:237] T 8f36eefcda3f475d9571e92fdeb52eb4 P 13d05ebdca8c4f489c597ce550dd773d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 2, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { replace: true } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: false } }
I20240501 07:59:21.392053 6444 raft_consensus.cc:1261] T 8f36eefcda3f475d9571e92fdeb52eb4 P 8315d22c56f7439a9e1fc82f153f018c [term 2 FOLLOWER]: Refusing update from remote peer 13d05ebdca8c4f489c597ce550dd773d: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 07:59:21.392158 6575 raft_consensus.cc:1261] T 8f36eefcda3f475d9571e92fdeb52eb4 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 2 LEARNER]: Refusing update from remote peer 13d05ebdca8c4f489c597ce550dd773d: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 07:59:21.392582 14130 consensus_queue.cc:1035] T 8f36eefcda3f475d9571e92fdeb52eb4 P 13d05ebdca8c4f489c597ce550dd773d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 07:59:21.393218 13418 consensus_queue.cc:1035] T 8f36eefcda3f475d9571e92fdeb52eb4 P 13d05ebdca8c4f489c597ce550dd773d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 07:59:21.393785 8859 consensus_peers.cc:485] T 8f36eefcda3f475d9571e92fdeb52eb4 P 13d05ebdca8c4f489c597ce550dd773d -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 07:59:21.395174 14130 raft_consensus.cc:2932] T 8f36eefcda3f475d9571e92fdeb52eb4 P 13d05ebdca8c4f489c597ce550dd773d [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 2 to 4, 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { replace: true } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: false } } }
I20240501 07:59:21.395620 6445 raft_consensus.cc:2932] T 8f36eefcda3f475d9571e92fdeb52eb4 P 8315d22c56f7439a9e1fc82f153f018c [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { replace: true } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: false } } }
I20240501 07:59:21.397909 6575 raft_consensus.cc:2932] T 8f36eefcda3f475d9571e92fdeb52eb4 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { replace: true } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: false } } }
I20240501 07:59:21.398640 6168 catalog_manager.cc:5484] T 8f36eefcda3f475d9571e92fdeb52eb4 P 13d05ebdca8c4f489c597ce550dd773d reported cstate change: config changed from index 2 to 4, 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "13d05ebdca8c4f489c597ce550dd773d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240501 07:59:21.520216 8859 consensus_peers.cc:485] T ff9dc384dbac47dc9795617c64688943 P 13d05ebdca8c4f489c597ce550dd773d -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 07:59:21.534796 6380 consensus_peers.cc:485] T 4664b04b23294ad4b0a0e5c87fdca2ad P 8315d22c56f7439a9e1fc82f153f018c -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 07:59:21.575078 6642 consensus_peers.cc:485] T 4bc915ffbbf345cbbe1195f8dd3ab874 P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 07:59:21.580565 6642 consensus_peers.cc:485] T 404bb81b9c404dd288d2f8160a4273da P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 07:59:21.657992 13961 consensus_queue.cc:1035] T d726533fb72d4827991e12034b42634e P 8315d22c56f7439a9e1fc82f153f018c [LEADER]: Connected to new peer: Peer: permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 07:59:21.662082 6380 consensus_peers.cc:485] T d726533fb72d4827991e12034b42634e P 8315d22c56f7439a9e1fc82f153f018c -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 07:59:21.695402 6511 consensus_peers.cc:462] T 0b24f67cf1a64debb23ee6618c820d33 P 3b164cbbc00f4e8eb3254ddd349a3ff9 -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111)
W20240501 07:59:21.722502 6380 consensus_peers.cc:485] T c3a9036ccfee45498bca9681518d3699 P 8315d22c56f7439a9e1fc82f153f018c -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 07:59:21.723284 6158 catalog_manager.cc:4573] TS a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): CreateTablet RPC failed for tablet 4664b04b23294ad4b0a0e5c87fdca2ad: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111)
W20240501 07:59:21.723582 8859 consensus_peers.cc:485] T c197ba256dc74e6d92e34799c274b4e3 P 13d05ebdca8c4f489c597ce550dd773d -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 07:59:21.743353 13961 consensus_queue.cc:1035] T 4664b04b23294ad4b0a0e5c87fdca2ad P 8315d22c56f7439a9e1fc82f153f018c [LEADER]: Connected to new peer: Peer: permanent_uuid: "13d05ebdca8c4f489c597ce550dd773d" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 07:59:21.770809 6380 consensus_peers.cc:485] T 782e37a4ebb34a68b461a74115ae8703 P 8315d22c56f7439a9e1fc82f153f018c -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 07:59:21.830263 8859 consensus_peers.cc:485] T fb6b46072a8d46e18b53b9f8de702859 P 13d05ebdca8c4f489c597ce550dd773d -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 07:59:21.865780 8859 consensus_peers.cc:485] T c197ba256dc74e6d92e34799c274b4e3 P 13d05ebdca8c4f489c597ce550dd773d -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 07:59:21.868364 6642 consensus_peers.cc:462] T a29720541b7b4a79aaa810592ef7c4b1 P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111)
W20240501 07:59:21.998016 8859 consensus_peers.cc:485] T f0c1c05110894e39b41e6b5c5894b8d8 P 13d05ebdca8c4f489c597ce550dd773d -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 07:59:22.002988 14049 raft_consensus.cc:482] T 4d4f47308d174a92bc73f23f2242be9a P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83f20e84af4c40d09a4804b6b26e697f)
I20240501 07:59:22.003116 14049 raft_consensus.cc:504] T 4d4f47308d174a92bc73f23f2242be9a P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { replace: true } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { promote: true } }
I20240501 07:59:22.003472 14049 leader_election.cc:288] T 4d4f47308d174a92bc73f23f2242be9a P 74077b61d6f54342bf9736b4a9f2eade [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149), a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607)
W20240501 07:59:22.004544 8728 leader_election.cc:334] T 4d4f47308d174a92bc73f23f2242be9a P 74077b61d6f54342bf9736b4a9f2eade [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111)
W20240501 07:59:22.004673 8728 leader_election.cc:334] T 4d4f47308d174a92bc73f23f2242be9a P 74077b61d6f54342bf9736b4a9f2eade [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111)
I20240501 07:59:22.004720 8728 leader_election.cc:302] T 4d4f47308d174a92bc73f23f2242be9a P 74077b61d6f54342bf9736b4a9f2eade [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74077b61d6f54342bf9736b4a9f2eade; no voters: 83f20e84af4c40d09a4804b6b26e697f, a505e7cb133c4392bfda8a4a20874ae9
I20240501 07:59:22.004873 14049 raft_consensus.cc:2726] T 4d4f47308d174a92bc73f23f2242be9a P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 07:59:22.006340 6249 consensus_peers.cc:485] T 5ee738571af740c1b494d4f17413d1d8 P 8b8b3ad3aa8f4053b2167dd550f2086f -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 07:59:22.017467 6380 consensus_peers.cc:485] T 782e37a4ebb34a68b461a74115ae8703 P 8315d22c56f7439a9e1fc82f153f018c -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 07:59:22.027864 6642 consensus_peers.cc:485] T 8454f24b0c28451b8ee51be39ca3e07f P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 07:59:22.031078 6249 consensus_peers.cc:485] T 6c3c0d6fe41f4ae196ae29beb9b68125 P 8b8b3ad3aa8f4053b2167dd550f2086f -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 07:59:22.048732 6249 consensus_peers.cc:485] T 0fc1ce00a2024e4a9c19abcc7d181efd P 8b8b3ad3aa8f4053b2167dd550f2086f -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 07:59:22.059159 8859 consensus_peers.cc:485] T bab84387a40d40c3b8af7f09eb462886 P 13d05ebdca8c4f489c597ce550dd773d -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 07:59:22.064007 8728 consensus_peers.cc:485] T f3375300a7914fa599fe0d3a8f1f9c18 P 74077b61d6f54342bf9736b4a9f2eade -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 07:59:22.076189 6642 consensus_peers.cc:485] T 3b9b71d44f754353b2321e8c36dd4dfd P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 07:59:22.076943 6511 consensus_peers.cc:462] T 0b24f67cf1a64debb23ee6618c820d33 P 3b164cbbc00f4e8eb3254ddd349a3ff9 -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111)
W20240501 07:59:22.078722 6511 consensus_peers.cc:485] T 114b6dbafa1c4c44ac42f1d2293e3125 P 3b164cbbc00f4e8eb3254ddd349a3ff9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 07:59:22.107239 6642 consensus_peers.cc:485] T a29720541b7b4a79aaa810592ef7c4b1 P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 07:59:22.109519 6249 consensus_peers.cc:485] T 2e540f389b2a4aef93a020f549d11b67 P 8b8b3ad3aa8f4053b2167dd550f2086f -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 07:59:22.122105 13961 raft_consensus.cc:482] T c86c420310ec42ffa5b24304fba74267 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83f20e84af4c40d09a4804b6b26e697f)
I20240501 07:59:22.122228 13961 raft_consensus.cc:504] T c86c420310ec42ffa5b24304fba74267 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { replace: true } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: true } }
I20240501 07:59:22.122617 13961 leader_election.cc:288] T c86c420310ec42ffa5b24304fba74267 P 8315d22c56f7439a9e1fc82f153f018c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149), a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607)
W20240501 07:59:22.123687 6380 leader_election.cc:334] T c86c420310ec42ffa5b24304fba74267 P 8315d22c56f7439a9e1fc82f153f018c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111)
W20240501 07:59:22.123796 6380 leader_election.cc:334] T c86c420310ec42ffa5b24304fba74267 P 8315d22c56f7439a9e1fc82f153f018c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111)
I20240501 07:59:22.123848 6380 leader_election.cc:302] T c86c420310ec42ffa5b24304fba74267 P 8315d22c56f7439a9e1fc82f153f018c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8315d22c56f7439a9e1fc82f153f018c; no voters: 83f20e84af4c40d09a4804b6b26e697f, a505e7cb133c4392bfda8a4a20874ae9
I20240501 07:59:22.123978 13961 raft_consensus.cc:2726] T c86c420310ec42ffa5b24304fba74267 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 07:59:22.145355 6511 consensus_peers.cc:485] T 6e2ac8c8857d45b88d3b51147d2609aa P 3b164cbbc00f4e8eb3254ddd349a3ff9 -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 07:59:22.189527 8728 consensus_peers.cc:485] T fcb3198c1fa5449781dc84756fe02bff P 74077b61d6f54342bf9736b4a9f2eade -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 07:59:22.210073 6511 consensus_peers.cc:485] T d6c28cd91f624673b8e428f471a1dcce P 3b164cbbc00f4e8eb3254ddd349a3ff9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 07:59:22.232153 8859 consensus_peers.cc:485] T 239ee596e62d47169eeb6670dc62d59c P 13d05ebdca8c4f489c597ce550dd773d -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 07:59:22.232779 6380 consensus_peers.cc:485] T bf806039d8c742d8a80ad994829bdf18 P 8315d22c56f7439a9e1fc82f153f018c -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 07:59:22.233387 8728 consensus_peers.cc:485] T ab1666f1e5204416adb435d9bbe80433 P 74077b61d6f54342bf9736b4a9f2eade -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 07:59:22.249352 6511 consensus_peers.cc:485] T 3745acee2b5543b697a3c8f45d2618bb P 3b164cbbc00f4e8eb3254ddd349a3ff9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 07:59:22.257216 13961 raft_consensus.cc:482] T 1de93edea7714491ae339db8cdfec090 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a505e7cb133c4392bfda8a4a20874ae9)
I20240501 07:59:22.257351 13961 raft_consensus.cc:504] T 1de93edea7714491ae339db8cdfec090 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } }
I20240501 07:59:22.257763 13961 leader_election.cc:288] T 1de93edea7714491ae339db8cdfec090 P 8315d22c56f7439a9e1fc82f153f018c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149), a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607)
W20240501 07:59:22.258967 6380 leader_election.cc:334] T 1de93edea7714491ae339db8cdfec090 P 8315d22c56f7439a9e1fc82f153f018c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111)
W20240501 07:59:22.259094 6380 leader_election.cc:334] T 1de93edea7714491ae339db8cdfec090 P 8315d22c56f7439a9e1fc82f153f018c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111)
I20240501 07:59:22.259155 6380 leader_election.cc:302] T 1de93edea7714491ae339db8cdfec090 P 8315d22c56f7439a9e1fc82f153f018c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8315d22c56f7439a9e1fc82f153f018c; no voters: 83f20e84af4c40d09a4804b6b26e697f, a505e7cb133c4392bfda8a4a20874ae9
I20240501 07:59:22.259327 13961 raft_consensus.cc:2726] T 1de93edea7714491ae339db8cdfec090 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 07:59:22.268412 13495 raft_consensus.cc:482] T 28eaf22342ff4b5394f958d0b659273c P 1f9cb72be98941b5a8e504cbe34132b9 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:22.268553 13495 raft_consensus.cc:504] T 28eaf22342ff4b5394f958d0b659273c P 1f9cb72be98941b5a8e504cbe34132b9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: false } }
I20240501 07:59:22.268985 13495 leader_election.cc:288] T 28eaf22342ff4b5394f958d0b659273c P 1f9cb72be98941b5a8e504cbe34132b9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607), 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131:34471)
I20240501 07:59:22.269419 6575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28eaf22342ff4b5394f958d0b659273c" candidate_uuid: "1f9cb72be98941b5a8e504cbe34132b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" is_pre_election: true
I20240501 07:59:22.269587 6575 raft_consensus.cc:2454] T 28eaf22342ff4b5394f958d0b659273c P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f9cb72be98941b5a8e504cbe34132b9 in term 2.
I20240501 07:59:22.269938 6644 leader_election.cc:302] T 28eaf22342ff4b5394f958d0b659273c P 1f9cb72be98941b5a8e504cbe34132b9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f9cb72be98941b5a8e504cbe34132b9, 3b164cbbc00f4e8eb3254ddd349a3ff9; no voters:
W20240501 07:59:22.270031 6642 leader_election.cc:334] T 28eaf22342ff4b5394f958d0b659273c P 1f9cb72be98941b5a8e504cbe34132b9 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111)
I20240501 07:59:22.270120 13495 raft_consensus.cc:2781] T 28eaf22342ff4b5394f958d0b659273c P 1f9cb72be98941b5a8e504cbe34132b9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240501 07:59:22.270169 13495 raft_consensus.cc:482] T 28eaf22342ff4b5394f958d0b659273c P 1f9cb72be98941b5a8e504cbe34132b9 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 07:59:22.270243 13495 raft_consensus.cc:3037] T 28eaf22342ff4b5394f958d0b659273c P 1f9cb72be98941b5a8e504cbe34132b9 [term 2 FOLLOWER]: Advancing to term 3
I20240501 07:59:22.271093 13495 raft_consensus.cc:504] T 28eaf22342ff4b5394f958d0b659273c P 1f9cb72be98941b5a8e504cbe34132b9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: false } }
I20240501 07:59:22.271417 13495 leader_election.cc:288] T 28eaf22342ff4b5394f958d0b659273c P 1f9cb72be98941b5a8e504cbe34132b9 [CANDIDATE]: Term 3 election: Requested vote from peers a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607), 3b164cbbc00f4e8eb3254ddd349a3ff9 (127.0.99.131:34471)
I20240501 07:59:22.271858 6575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28eaf22342ff4b5394f958d0b659273c" candidate_uuid: "1f9cb72be98941b5a8e504cbe34132b9" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9"
I20240501 07:59:22.271997 6575 raft_consensus.cc:3037] T 28eaf22342ff4b5394f958d0b659273c P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 2 FOLLOWER]: Advancing to term 3
W20240501 07:59:22.272361 6642 leader_election.cc:334] T 28eaf22342ff4b5394f958d0b659273c P 1f9cb72be98941b5a8e504cbe34132b9 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111)
I20240501 07:59:22.273087 6575 raft_consensus.cc:2454] T 28eaf22342ff4b5394f958d0b659273c P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f9cb72be98941b5a8e504cbe34132b9 in term 3.
I20240501 07:59:22.273403 6644 leader_election.cc:302] T 28eaf22342ff4b5394f958d0b659273c P 1f9cb72be98941b5a8e504cbe34132b9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1f9cb72be98941b5a8e504cbe34132b9, 3b164cbbc00f4e8eb3254ddd349a3ff9; no voters: a505e7cb133c4392bfda8a4a20874ae9
I20240501 07:59:22.273550 13495 raft_consensus.cc:2781] T 28eaf22342ff4b5394f958d0b659273c P 1f9cb72be98941b5a8e504cbe34132b9 [term 3 FOLLOWER]: Leader election won for term 3
I20240501 07:59:22.273687 13495 raft_consensus.cc:686] T 28eaf22342ff4b5394f958d0b659273c P 1f9cb72be98941b5a8e504cbe34132b9 [term 3 LEADER]: Becoming Leader. State: Replica: 1f9cb72be98941b5a8e504cbe34132b9, State: Running, Role: LEADER
I20240501 07:59:22.273849 13495 consensus_queue.cc:237] T 28eaf22342ff4b5394f958d0b659273c P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: false } }
I20240501 07:59:22.275848 6164 catalog_manager.cc:5484] T 28eaf22342ff4b5394f958d0b659273c P 1f9cb72be98941b5a8e504cbe34132b9 reported cstate change: term changed from 1 to 3, leader changed from a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135) to 1f9cb72be98941b5a8e504cbe34132b9 (127.0.99.132). New cstate: current_term: 3 leader_uuid: "1f9cb72be98941b5a8e504cbe34132b9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f9cb72be98941b5a8e504cbe34132b9" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36773 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20240501 07:59:22.292986 8728 consensus_peers.cc:485] T 6dc929e2a6ae45bd8b1f297583636bb6 P 74077b61d6f54342bf9736b4a9f2eade -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 07:59:22.316531 6380 consensus_peers.cc:485] T cfcc814f4d2d488ba745ecaf0cc6d61a P 8315d22c56f7439a9e1fc82f153f018c -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 07:59:22.320303 6642 consensus_peers.cc:485] T 6daebcfc41754ec4aca939ca20588854 P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 07:59:22.333340 6380 consensus_peers.cc:485] T a756205204d146ce82a2799e30582f57 P 8315d22c56f7439a9e1fc82f153f018c -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 07:59:22.338769 14049 raft_consensus.cc:482] T a93ce5841c7f40a9ab23f88efe7b85bb P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83f20e84af4c40d09a4804b6b26e697f)
I20240501 07:59:22.338917 14049 raft_consensus.cc:504] T a93ce5841c7f40a9ab23f88efe7b85bb P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { replace: true } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { promote: true } }
I20240501 07:59:22.339345 14049 leader_election.cc:288] T a93ce5841c7f40a9ab23f88efe7b85bb P 74077b61d6f54342bf9736b4a9f2eade [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149), a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607)
W20240501 07:59:22.340268 8728 leader_election.cc:334] T a93ce5841c7f40a9ab23f88efe7b85bb P 74077b61d6f54342bf9736b4a9f2eade [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111)
W20240501 07:59:22.340363 8728 leader_election.cc:334] T a93ce5841c7f40a9ab23f88efe7b85bb P 74077b61d6f54342bf9736b4a9f2eade [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111)
I20240501 07:59:22.340397 8728 leader_election.cc:302] T a93ce5841c7f40a9ab23f88efe7b85bb P 74077b61d6f54342bf9736b4a9f2eade [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74077b61d6f54342bf9736b4a9f2eade; no voters: 83f20e84af4c40d09a4804b6b26e697f, a505e7cb133c4392bfda8a4a20874ae9
I20240501 07:59:22.340533 14175 raft_consensus.cc:2726] T a93ce5841c7f40a9ab23f88efe7b85bb P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 07:59:22.422194 8859 consensus_peers.cc:485] T e0583fa43994452ca7553cbf2309eba1 P 13d05ebdca8c4f489c597ce550dd773d -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 07:59:22.440364 8728 consensus_peers.cc:485] T 36aaf192042f4320b2526322424dc640 P 74077b61d6f54342bf9736b4a9f2eade -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 07:59:22.451893 6380 consensus_peers.cc:485] T bf806039d8c742d8a80ad994829bdf18 P 8315d22c56f7439a9e1fc82f153f018c -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 07:59:22.470815 8859 consensus_peers.cc:485] T ad3e528eaff349a4b3b03037652b1505 P 13d05ebdca8c4f489c597ce550dd773d -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 07:59:22.472064 14158 raft_consensus.cc:482] T 2976d6353e8947b7b6c37f72efbeeac4 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83f20e84af4c40d09a4804b6b26e697f)
I20240501 07:59:22.472496 14158 raft_consensus.cc:504] T 2976d6353e8947b7b6c37f72efbeeac4 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: false } }
I20240501 07:59:22.472990 14158 leader_election.cc:288] T 2976d6353e8947b7b6c37f72efbeeac4 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607), 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149)
W20240501 07:59:22.473985 6511 leader_election.cc:334] T 2976d6353e8947b7b6c37f72efbeeac4 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111)
W20240501 07:59:22.474450 6511 leader_election.cc:334] T 2976d6353e8947b7b6c37f72efbeeac4 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111)
I20240501 07:59:22.474514 6511 leader_election.cc:302] T 2976d6353e8947b7b6c37f72efbeeac4 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b164cbbc00f4e8eb3254ddd349a3ff9; no voters: 83f20e84af4c40d09a4804b6b26e697f, a505e7cb133c4392bfda8a4a20874ae9
I20240501 07:59:22.474686 14176 raft_consensus.cc:2726] T 2976d6353e8947b7b6c37f72efbeeac4 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 07:59:22.475946 6642 consensus_peers.cc:462] T a29720541b7b4a79aaa810592ef7c4b1 P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111)
W20240501 07:59:22.496855 8859 consensus_peers.cc:485] T e0583fa43994452ca7553cbf2309eba1 P 13d05ebdca8c4f489c597ce550dd773d -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 07:59:22.506873 6511 consensus_peers.cc:485] T d3f2250a62c6429e886753aaffcbe883 P 3b164cbbc00f4e8eb3254ddd349a3ff9 -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 07:59:22.532889 6249 consensus_peers.cc:485] T f68dbc1f65eb4f1b91a62e2de82a7676 P 8b8b3ad3aa8f4053b2167dd550f2086f -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 07:59:22.576613 6380 consensus_peers.cc:485] T 9461ad0f65e1489ca102f2558194ce8f P 8315d22c56f7439a9e1fc82f153f018c -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 07:59:22.666500 6511 consensus_peers.cc:462] T 0b24f67cf1a64debb23ee6618c820d33 P 3b164cbbc00f4e8eb3254ddd349a3ff9 -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111)
I20240501 07:59:22.674633 13961 raft_consensus.cc:482] T 13daf8496cd64d7d959ce5ad666b9be8 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a505e7cb133c4392bfda8a4a20874ae9)
I20240501 07:59:22.674757 13961 raft_consensus.cc:504] T 13daf8496cd64d7d959ce5ad666b9be8 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } }
I20240501 07:59:22.675112 13961 leader_election.cc:288] T 13daf8496cd64d7d959ce5ad666b9be8 P 8315d22c56f7439a9e1fc82f153f018c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149), a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607)
W20240501 07:59:22.676173 6380 leader_election.cc:334] T 13daf8496cd64d7d959ce5ad666b9be8 P 8315d22c56f7439a9e1fc82f153f018c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111)
W20240501 07:59:22.676295 6380 leader_election.cc:334] T 13daf8496cd64d7d959ce5ad666b9be8 P 8315d22c56f7439a9e1fc82f153f018c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111)
I20240501 07:59:22.676348 6380 leader_election.cc:302] T 13daf8496cd64d7d959ce5ad666b9be8 P 8315d22c56f7439a9e1fc82f153f018c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8315d22c56f7439a9e1fc82f153f018c; no voters: 83f20e84af4c40d09a4804b6b26e697f, a505e7cb133c4392bfda8a4a20874ae9
I20240501 07:59:22.676486 13961 raft_consensus.cc:2726] T 13daf8496cd64d7d959ce5ad666b9be8 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 07:59:22.709223 6511 consensus_peers.cc:485] T d3f2250a62c6429e886753aaffcbe883 P 3b164cbbc00f4e8eb3254ddd349a3ff9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 07:59:22.728783 6642 consensus_peers.cc:485] T 28eaf22342ff4b5394f958d0b659273c P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 07:59:22.752156 6249 consensus_peers.cc:485] T 1c32cd00bc344ae1b161d3f9b43b53d3 P 8b8b3ad3aa8f4053b2167dd550f2086f -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 07:59:22.752756 6164 catalog_manager.cc:5576] TS 13d05ebdca8c4f489c597ce550dd773d (127.0.99.134:40141) does not have the latest schema for tablet c197ba256dc74e6d92e34799c274b4e3 (table rebalancer_dynamic_table [id=158b84b3e18943d38616a2fb6e606d35]). Expected version 1 got 0
W20240501 07:59:22.764600 8859 consensus_peers.cc:485] T 0864c290fafa488abefa93d41c0a2efe P 13d05ebdca8c4f489c597ce550dd773d -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 07:59:22.857003 6575 raft_consensus.cc:1261] T 28eaf22342ff4b5394f958d0b659273c P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 3 FOLLOWER]: Refusing update from remote peer 1f9cb72be98941b5a8e504cbe34132b9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240501 07:59:22.857506 14170 consensus_queue.cc:1035] T 28eaf22342ff4b5394f958d0b659273c P 1f9cb72be98941b5a8e504cbe34132b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 07:59:22.887871 14176 raft_consensus.cc:482] T e81b23661a1d475887c63f79d8f90ab6 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:22.888000 14176 raft_consensus.cc:504] T e81b23661a1d475887c63f79d8f90ab6 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } }
I20240501 07:59:22.888396 14176 leader_election.cc:288] T e81b23661a1d475887c63f79d8f90ab6 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149), a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607)
W20240501 07:59:22.889545 6511 leader_election.cc:334] T e81b23661a1d475887c63f79d8f90ab6 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111)
W20240501 07:59:22.889683 6511 leader_election.cc:334] T e81b23661a1d475887c63f79d8f90ab6 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111)
I20240501 07:59:22.889726 6511 leader_election.cc:302] T e81b23661a1d475887c63f79d8f90ab6 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b164cbbc00f4e8eb3254ddd349a3ff9; no voters: 83f20e84af4c40d09a4804b6b26e697f, a505e7cb133c4392bfda8a4a20874ae9
I20240501 07:59:22.889853 14176 raft_consensus.cc:2726] T e81b23661a1d475887c63f79d8f90ab6 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240501 07:59:22.898093 8728 consensus_peers.cc:485] T 0f9e2549fe294a93b3b25081a15b2028 P 74077b61d6f54342bf9736b4a9f2eade -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 07:59:22.940903 8728 consensus_peers.cc:485] T c5a876dfb598428ab350593129505504 P 74077b61d6f54342bf9736b4a9f2eade -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 07:59:23.038043 6642 consensus_peers.cc:462] T a29720541b7b4a79aaa810592ef7c4b1 P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111)
I20240501 07:59:23.070791 14175 raft_consensus.cc:482] T 7dbbfc5d5848486f983cffdd2f36cc91 P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:23.070914 14175 raft_consensus.cc:504] T 7dbbfc5d5848486f983cffdd2f36cc91 P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } }
I20240501 07:59:23.071255 14175 leader_election.cc:288] T 7dbbfc5d5848486f983cffdd2f36cc91 P 74077b61d6f54342bf9736b4a9f2eade [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149), a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607)
W20240501 07:59:23.072209 8728 leader_election.cc:334] T 7dbbfc5d5848486f983cffdd2f36cc91 P 74077b61d6f54342bf9736b4a9f2eade [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111)
W20240501 07:59:23.072331 8728 leader_election.cc:334] T 7dbbfc5d5848486f983cffdd2f36cc91 P 74077b61d6f54342bf9736b4a9f2eade [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111)
I20240501 07:59:23.072384 8728 leader_election.cc:302] T 7dbbfc5d5848486f983cffdd2f36cc91 P 74077b61d6f54342bf9736b4a9f2eade [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74077b61d6f54342bf9736b4a9f2eade; no voters: 83f20e84af4c40d09a4804b6b26e697f, a505e7cb133c4392bfda8a4a20874ae9
I20240501 07:59:23.072499 14175 raft_consensus.cc:2726] T 7dbbfc5d5848486f983cffdd2f36cc91 P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 07:59:23.165788 8859 consensus_peers.cc:485] T b4a2db0198774bb2b4f3df769260e32f P 13d05ebdca8c4f489c597ce550dd773d -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 07:59:23.254818 6511 consensus_peers.cc:462] T 0b24f67cf1a64debb23ee6618c820d33 P 3b164cbbc00f4e8eb3254ddd349a3ff9 -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111)
W20240501 07:59:23.273646 6158 catalog_manager.cc:4573] TS a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): CreateTablet RPC failed for tablet e81b23661a1d475887c63f79d8f90ab6: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111)
W20240501 07:59:23.318608 8728 consensus_peers.cc:485] T e5bd3a8f7a7640c09774f781009eb19c P 74077b61d6f54342bf9736b4a9f2eade -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 07:59:23.402664 6511 consensus_peers.cc:485] T af718dc43add4edfac703013655699f4 P 3b164cbbc00f4e8eb3254ddd349a3ff9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 07:59:23.404498 8859 consensus_peers.cc:485] T c16b7962c18d4e02b850c7c7c678eae4 P 13d05ebdca8c4f489c597ce550dd773d -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 07:59:23.408174 8859 consensus_peers.cc:485] T 8a39f168c8f846cd9b8c4e2f2cf62fdd P 13d05ebdca8c4f489c597ce550dd773d -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 07:59:23.453456 8859 consensus_peers.cc:485] T f788de4df5454d00ac9bb883f0abc9bd P 13d05ebdca8c4f489c597ce550dd773d -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 07:59:23.530858 6380 consensus_peers.cc:485] T f88fe033bce14bf0b1b63a52a7ba137e P 8315d22c56f7439a9e1fc82f153f018c -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 07:59:23.544713 8859 consensus_peers.cc:485] T 12ce6a21d06c4ba889542758be9dc62d P 13d05ebdca8c4f489c597ce550dd773d -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 07:59:23.594843 6642 consensus_peers.cc:462] T a29720541b7b4a79aaa810592ef7c4b1 P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111)
W20240501 07:59:23.633535 6380 consensus_peers.cc:485] T 72aa4dc7664b48b7a73d34feba7d44d3 P 8315d22c56f7439a9e1fc82f153f018c -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 07:59:23.642139 6249 consensus_peers.cc:485] T b7d57b7fd5234f50afbc859bc997f3e1 P 8b8b3ad3aa8f4053b2167dd550f2086f -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 07:59:23.653841 6511 consensus_peers.cc:462] T 0b24f67cf1a64debb23ee6618c820d33 P 3b164cbbc00f4e8eb3254ddd349a3ff9 -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111)
W20240501 07:59:23.795146 6642 consensus_peers.cc:485] T 2560b1ec81ac4357b32c7a8ba9d72216 P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 07:59:23.895392 8859 consensus_peers.cc:485] T ff9dc384dbac47dc9795617c64688943 P 13d05ebdca8c4f489c597ce550dd773d -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 07:59:23.998454 8859 consensus_peers.cc:485] T 8f36eefcda3f475d9571e92fdeb52eb4 P 13d05ebdca8c4f489c597ce550dd773d -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 07:59:24.031181 8859 consensus_peers.cc:485] T fb6b46072a8d46e18b53b9f8de702859 P 13d05ebdca8c4f489c597ce550dd773d -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 07:59:24.094069 8859 consensus_peers.cc:485] T c197ba256dc74e6d92e34799c274b4e3 P 13d05ebdca8c4f489c597ce550dd773d -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 07:59:24.116644 14175 raft_consensus.cc:482] T 4d4f47308d174a92bc73f23f2242be9a P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83f20e84af4c40d09a4804b6b26e697f)
I20240501 07:59:24.116817 14175 raft_consensus.cc:504] T 4d4f47308d174a92bc73f23f2242be9a P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { replace: true } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { promote: true } }
W20240501 07:59:24.116930 6642 consensus_peers.cc:462] T a29720541b7b4a79aaa810592ef7c4b1 P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111)
I20240501 07:59:24.117314 14175 leader_election.cc:288] T 4d4f47308d174a92bc73f23f2242be9a P 74077b61d6f54342bf9736b4a9f2eade [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149), a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607)
W20240501 07:59:24.118463 8728 leader_election.cc:334] T 4d4f47308d174a92bc73f23f2242be9a P 74077b61d6f54342bf9736b4a9f2eade [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111)
W20240501 07:59:24.118585 8728 leader_election.cc:334] T 4d4f47308d174a92bc73f23f2242be9a P 74077b61d6f54342bf9736b4a9f2eade [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111)
I20240501 07:59:24.118636 8728 leader_election.cc:302] T 4d4f47308d174a92bc73f23f2242be9a P 74077b61d6f54342bf9736b4a9f2eade [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74077b61d6f54342bf9736b4a9f2eade; no voters: 83f20e84af4c40d09a4804b6b26e697f, a505e7cb133c4392bfda8a4a20874ae9
I20240501 07:59:24.118814 14175 raft_consensus.cc:2726] T 4d4f47308d174a92bc73f23f2242be9a P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 07:59:24.133412 6380 consensus_peers.cc:485] T c3a9036ccfee45498bca9681518d3699 P 8315d22c56f7439a9e1fc82f153f018c -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 07:59:24.134884 6511 consensus_peers.cc:462] T 0b24f67cf1a64debb23ee6618c820d33 P 3b164cbbc00f4e8eb3254ddd349a3ff9 -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111)
W20240501 07:59:24.145819 6642 consensus_peers.cc:485] T 404bb81b9c404dd288d2f8160a4273da P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 07:59:24.184260 6642 consensus_peers.cc:485] T a29720541b7b4a79aaa810592ef7c4b1 P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 07:59:24.194705 6380 consensus_peers.cc:485] T 4664b04b23294ad4b0a0e5c87fdca2ad P 8315d22c56f7439a9e1fc82f153f018c -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 07:59:24.209132 6511 consensus_peers.cc:485] T 6e2ac8c8857d45b88d3b51147d2609aa P 3b164cbbc00f4e8eb3254ddd349a3ff9 -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 07:59:24.217526 6249 consensus_peers.cc:485] T 5ee738571af740c1b494d4f17413d1d8 P 8b8b3ad3aa8f4053b2167dd550f2086f -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 07:59:24.245865 6380 consensus_peers.cc:485] T d726533fb72d4827991e12034b42634e P 8315d22c56f7439a9e1fc82f153f018c -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 07:59:24.301419 6642 consensus_peers.cc:485] T 8454f24b0c28451b8ee51be39ca3e07f P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 07:59:24.344244 6380 consensus_peers.cc:485] T 782e37a4ebb34a68b461a74115ae8703 P 8315d22c56f7439a9e1fc82f153f018c -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 07:59:24.355324 6642 consensus_peers.cc:485] T 3b9b71d44f754353b2321e8c36dd4dfd P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 07:59:24.372887 6158 catalog_manager.cc:4573] TS 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 144ff11d2a504f7ab7a230e6822d24ca: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111)
W20240501 07:59:24.429064 6642 consensus_peers.cc:485] T 4bc915ffbbf345cbbe1195f8dd3ab874 P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 07:59:24.470846 6249 consensus_peers.cc:485] T 0fc1ce00a2024e4a9c19abcc7d181efd P 8b8b3ad3aa8f4053b2167dd550f2086f -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 07:59:24.485949 6380 consensus_peers.cc:485] T 782e37a4ebb34a68b461a74115ae8703 P 8315d22c56f7439a9e1fc82f153f018c -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 07:59:24.503930 8859 consensus_peers.cc:485] T c197ba256dc74e6d92e34799c274b4e3 P 13d05ebdca8c4f489c597ce550dd773d -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 07:59:24.508203 6511 consensus_peers.cc:485] T 114b6dbafa1c4c44ac42f1d2293e3125 P 3b164cbbc00f4e8eb3254ddd349a3ff9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 07:59:24.534546 6380 consensus_peers.cc:485] T bf806039d8c742d8a80ad994829bdf18 P 8315d22c56f7439a9e1fc82f153f018c -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 07:59:24.560189 8859 consensus_peers.cc:485] T 239ee596e62d47169eeb6670dc62d59c P 13d05ebdca8c4f489c597ce550dd773d -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 07:59:24.560736 6642 consensus_peers.cc:462] T a29720541b7b4a79aaa810592ef7c4b1 P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111)
W20240501 07:59:24.568499 6249 consensus_peers.cc:485] T 6c3c0d6fe41f4ae196ae29beb9b68125 P 8b8b3ad3aa8f4053b2167dd550f2086f -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 07:59:24.576368 6511 consensus_peers.cc:462] T 0b24f67cf1a64debb23ee6618c820d33 P 3b164cbbc00f4e8eb3254ddd349a3ff9 -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111)
W20240501 07:59:24.652519 6249 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111) [suppressed 72 similar messages]
W20240501 07:59:24.654392 8859 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111) [suppressed 189 similar messages]
W20240501 07:59:24.655107 8859 consensus_peers.cc:485] T e0583fa43994452ca7553cbf2309eba1 P 13d05ebdca8c4f489c597ce550dd773d -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 07:59:24.663702 8859 consensus_peers.cc:485] T bab84387a40d40c3b8af7f09eb462886 P 13d05ebdca8c4f489c597ce550dd773d -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 07:59:24.680302 14195 raft_consensus.cc:482] T c86c420310ec42ffa5b24304fba74267 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83f20e84af4c40d09a4804b6b26e697f)
I20240501 07:59:24.680441 14195 raft_consensus.cc:504] T c86c420310ec42ffa5b24304fba74267 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { replace: true } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: true } }
I20240501 07:59:24.680804 14195 leader_election.cc:288] T c86c420310ec42ffa5b24304fba74267 P 8315d22c56f7439a9e1fc82f153f018c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149), a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607)
W20240501 07:59:24.681397 6380 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111) [suppressed 133 similar messages]
W20240501 07:59:24.682001 6380 leader_election.cc:334] T c86c420310ec42ffa5b24304fba74267 P 8315d22c56f7439a9e1fc82f153f018c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111)
W20240501 07:59:24.682144 6380 leader_election.cc:334] T c86c420310ec42ffa5b24304fba74267 P 8315d22c56f7439a9e1fc82f153f018c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111)
I20240501 07:59:24.682193 6380 leader_election.cc:302] T c86c420310ec42ffa5b24304fba74267 P 8315d22c56f7439a9e1fc82f153f018c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8315d22c56f7439a9e1fc82f153f018c; no voters: 83f20e84af4c40d09a4804b6b26e697f, a505e7cb133c4392bfda8a4a20874ae9
I20240501 07:59:24.682343 14195 raft_consensus.cc:2726] T c86c420310ec42ffa5b24304fba74267 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 07:59:24.710981 6511 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111) [suppressed 101 similar messages]
W20240501 07:59:24.711589 6511 consensus_peers.cc:485] T d6c28cd91f624673b8e428f471a1dcce P 3b164cbbc00f4e8eb3254ddd349a3ff9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 07:59:24.726634 6642 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111) [suppressed 93 similar messages]
W20240501 07:59:24.734289 6380 consensus_peers.cc:485] T bf806039d8c742d8a80ad994829bdf18 P 8315d22c56f7439a9e1fc82f153f018c -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 07:59:24.734855 8728 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111) [suppressed 100 similar messages]
W20240501 07:59:24.735404 8728 consensus_peers.cc:485] T ab1666f1e5204416adb435d9bbe80433 P 74077b61d6f54342bf9736b4a9f2eade -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 07:59:24.773545 14175 raft_consensus.cc:482] T a93ce5841c7f40a9ab23f88efe7b85bb P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83f20e84af4c40d09a4804b6b26e697f)
I20240501 07:59:24.773712 14175 raft_consensus.cc:504] T a93ce5841c7f40a9ab23f88efe7b85bb P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { replace: true } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { promote: true } }
I20240501 07:59:24.774189 14175 leader_election.cc:288] T a93ce5841c7f40a9ab23f88efe7b85bb P 74077b61d6f54342bf9736b4a9f2eade [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149), a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607)
W20240501 07:59:24.774507 6511 consensus_peers.cc:485] T 3745acee2b5543b697a3c8f45d2618bb P 3b164cbbc00f4e8eb3254ddd349a3ff9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 07:59:24.775305 8728 leader_election.cc:334] T a93ce5841c7f40a9ab23f88efe7b85bb P 74077b61d6f54342bf9736b4a9f2eade [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111)
W20240501 07:59:24.775410 8728 leader_election.cc:334] T a93ce5841c7f40a9ab23f88efe7b85bb P 74077b61d6f54342bf9736b4a9f2eade [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111)
I20240501 07:59:24.775460 8728 leader_election.cc:302] T a93ce5841c7f40a9ab23f88efe7b85bb P 74077b61d6f54342bf9736b4a9f2eade [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74077b61d6f54342bf9736b4a9f2eade; no voters: 83f20e84af4c40d09a4804b6b26e697f, a505e7cb133c4392bfda8a4a20874ae9
I20240501 07:59:24.775616 14175 raft_consensus.cc:2726] T a93ce5841c7f40a9ab23f88efe7b85bb P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 07:59:24.797989 6249 consensus_peers.cc:485] T 2e540f389b2a4aef93a020f549d11b67 P 8b8b3ad3aa8f4053b2167dd550f2086f -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 07:59:24.821806 8728 consensus_peers.cc:485] T f3375300a7914fa599fe0d3a8f1f9c18 P 74077b61d6f54342bf9736b4a9f2eade -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 07:59:24.831058 6642 consensus_peers.cc:485] T 6daebcfc41754ec4aca939ca20588854 P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 07:59:24.842358 6380 consensus_peers.cc:485] T cfcc814f4d2d488ba745ecaf0cc6d61a P 8315d22c56f7439a9e1fc82f153f018c -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 07:59:24.865379 6249 consensus_peers.cc:485] T f68dbc1f65eb4f1b91a62e2de82a7676 P 8b8b3ad3aa8f4053b2167dd550f2086f -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 07:59:24.887420 8859 consensus_peers.cc:485] T e0583fa43994452ca7553cbf2309eba1 P 13d05ebdca8c4f489c597ce550dd773d -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 07:59:24.894851 8859 consensus_peers.cc:485] T f0c1c05110894e39b41e6b5c5894b8d8 P 13d05ebdca8c4f489c597ce550dd773d -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 07:59:24.906404 6511 consensus_peers.cc:485] T d3f2250a62c6429e886753aaffcbe883 P 3b164cbbc00f4e8eb3254ddd349a3ff9 -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 07:59:24.931617 14195 raft_consensus.cc:482] T 1de93edea7714491ae339db8cdfec090 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a505e7cb133c4392bfda8a4a20874ae9)
I20240501 07:59:24.931744 14195 raft_consensus.cc:504] T 1de93edea7714491ae339db8cdfec090 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } }
I20240501 07:59:24.932179 14195 leader_election.cc:288] T 1de93edea7714491ae339db8cdfec090 P 8315d22c56f7439a9e1fc82f153f018c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149), a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607)
W20240501 07:59:24.933331 6380 leader_election.cc:334] T 1de93edea7714491ae339db8cdfec090 P 8315d22c56f7439a9e1fc82f153f018c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111)
W20240501 07:59:24.933446 6380 leader_election.cc:334] T 1de93edea7714491ae339db8cdfec090 P 8315d22c56f7439a9e1fc82f153f018c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111)
I20240501 07:59:24.933512 6380 leader_election.cc:302] T 1de93edea7714491ae339db8cdfec090 P 8315d22c56f7439a9e1fc82f153f018c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8315d22c56f7439a9e1fc82f153f018c; no voters: 83f20e84af4c40d09a4804b6b26e697f, a505e7cb133c4392bfda8a4a20874ae9
I20240501 07:59:24.933666 14195 raft_consensus.cc:2726] T 1de93edea7714491ae339db8cdfec090 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 07:59:24.959597 8728 consensus_peers.cc:485] T 6dc929e2a6ae45bd8b1f297583636bb6 P 74077b61d6f54342bf9736b4a9f2eade -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 07:59:24.962693 8728 consensus_peers.cc:485] T fcb3198c1fa5449781dc84756fe02bff P 74077b61d6f54342bf9736b4a9f2eade -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 07:59:25.025236 6511 consensus_peers.cc:462] T 0b24f67cf1a64debb23ee6618c820d33 P 3b164cbbc00f4e8eb3254ddd349a3ff9 -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111)
W20240501 07:59:25.027196 6380 consensus_peers.cc:485] T 9461ad0f65e1489ca102f2558194ce8f P 8315d22c56f7439a9e1fc82f153f018c -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 07:59:25.059763 6380 consensus_peers.cc:485] T a756205204d146ce82a2799e30582f57 P 8315d22c56f7439a9e1fc82f153f018c -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 07:59:25.066112 6511 consensus_peers.cc:485] T d3f2250a62c6429e886753aaffcbe883 P 3b164cbbc00f4e8eb3254ddd349a3ff9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 07:59:25.070088 8728 consensus_peers.cc:485] T 36aaf192042f4320b2526322424dc640 P 74077b61d6f54342bf9736b4a9f2eade -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 07:59:25.088588 8859 consensus_peers.cc:485] T ad3e528eaff349a4b3b03037652b1505 P 13d05ebdca8c4f489c597ce550dd773d -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 07:59:25.121405 14175 raft_consensus.cc:482] T 7dbbfc5d5848486f983cffdd2f36cc91 P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:25.121551 14175 raft_consensus.cc:504] T 7dbbfc5d5848486f983cffdd2f36cc91 P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } }
W20240501 07:59:25.121619 6642 consensus_peers.cc:462] T a29720541b7b4a79aaa810592ef7c4b1 P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111)
I20240501 07:59:25.121940 14175 leader_election.cc:288] T 7dbbfc5d5848486f983cffdd2f36cc91 P 74077b61d6f54342bf9736b4a9f2eade [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149), a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607)
W20240501 07:59:25.123045 8728 leader_election.cc:334] T 7dbbfc5d5848486f983cffdd2f36cc91 P 74077b61d6f54342bf9736b4a9f2eade [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111)
W20240501 07:59:25.123157 8728 leader_election.cc:334] T 7dbbfc5d5848486f983cffdd2f36cc91 P 74077b61d6f54342bf9736b4a9f2eade [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111)
I20240501 07:59:25.123201 8728 leader_election.cc:302] T 7dbbfc5d5848486f983cffdd2f36cc91 P 74077b61d6f54342bf9736b4a9f2eade [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74077b61d6f54342bf9736b4a9f2eade; no voters: 83f20e84af4c40d09a4804b6b26e697f, a505e7cb133c4392bfda8a4a20874ae9
I20240501 07:59:25.123339 14175 raft_consensus.cc:2726] T 7dbbfc5d5848486f983cffdd2f36cc91 P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 07:59:25.160266 6642 consensus_peers.cc:485] T 28eaf22342ff4b5394f958d0b659273c P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 07:59:25.184389 8728 consensus_peers.cc:485] T c5a876dfb598428ab350593129505504 P 74077b61d6f54342bf9736b4a9f2eade -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 07:59:25.329990 6158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111) [suppressed 76 similar messages]
W20240501 07:59:25.384121 6158 catalog_manager.cc:4573] TS 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): CreateTablet RPC failed for tablet e81b23661a1d475887c63f79d8f90ab6: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111)
W20240501 07:59:25.434281 6511 consensus_peers.cc:462] T 0b24f67cf1a64debb23ee6618c820d33 P 3b164cbbc00f4e8eb3254ddd349a3ff9 -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111)
W20240501 07:59:25.491994 6249 consensus_peers.cc:485] T 1c32cd00bc344ae1b161d3f9b43b53d3 P 8b8b3ad3aa8f4053b2167dd550f2086f -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 07:59:25.569875 8859 consensus_peers.cc:485] T 0864c290fafa488abefa93d41c0a2efe P 13d05ebdca8c4f489c597ce550dd773d -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 07:59:25.622746 8728 consensus_peers.cc:485] T e5bd3a8f7a7640c09774f781009eb19c P 74077b61d6f54342bf9736b4a9f2eade -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 07:59:25.665776 8859 consensus_peers.cc:485] T f788de4df5454d00ac9bb883f0abc9bd P 13d05ebdca8c4f489c597ce550dd773d -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 07:59:25.701803 14202 raft_consensus.cc:482] T e81b23661a1d475887c63f79d8f90ab6 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:25.701941 14202 raft_consensus.cc:504] T e81b23661a1d475887c63f79d8f90ab6 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } }
I20240501 07:59:25.702386 14202 leader_election.cc:288] T e81b23661a1d475887c63f79d8f90ab6 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149), a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607)
W20240501 07:59:25.703593 6642 consensus_peers.cc:462] T a29720541b7b4a79aaa810592ef7c4b1 P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111)
W20240501 07:59:25.703747 6511 leader_election.cc:334] T e81b23661a1d475887c63f79d8f90ab6 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111)
W20240501 07:59:25.703856 6511 leader_election.cc:334] T e81b23661a1d475887c63f79d8f90ab6 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111)
I20240501 07:59:25.703903 6511 leader_election.cc:302] T e81b23661a1d475887c63f79d8f90ab6 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b164cbbc00f4e8eb3254ddd349a3ff9; no voters: 83f20e84af4c40d09a4804b6b26e697f, a505e7cb133c4392bfda8a4a20874ae9
I20240501 07:59:25.704049 14202 raft_consensus.cc:2726] T e81b23661a1d475887c63f79d8f90ab6 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 07:59:25.732218 14195 raft_consensus.cc:482] T 13daf8496cd64d7d959ce5ad666b9be8 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a505e7cb133c4392bfda8a4a20874ae9)
I20240501 07:59:25.732357 14195 raft_consensus.cc:504] T 13daf8496cd64d7d959ce5ad666b9be8 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } }
I20240501 07:59:25.732740 14195 leader_election.cc:288] T 13daf8496cd64d7d959ce5ad666b9be8 P 8315d22c56f7439a9e1fc82f153f018c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149), a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607)
W20240501 07:59:25.733779 6380 leader_election.cc:334] T 13daf8496cd64d7d959ce5ad666b9be8 P 8315d22c56f7439a9e1fc82f153f018c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111)
W20240501 07:59:25.733907 6380 leader_election.cc:334] T 13daf8496cd64d7d959ce5ad666b9be8 P 8315d22c56f7439a9e1fc82f153f018c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111)
I20240501 07:59:25.733968 6380 leader_election.cc:302] T 13daf8496cd64d7d959ce5ad666b9be8 P 8315d22c56f7439a9e1fc82f153f018c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8315d22c56f7439a9e1fc82f153f018c; no voters: 83f20e84af4c40d09a4804b6b26e697f, a505e7cb133c4392bfda8a4a20874ae9
I20240501 07:59:25.734097 14195 raft_consensus.cc:2726] T 13daf8496cd64d7d959ce5ad666b9be8 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 07:59:25.742719 8859 consensus_peers.cc:485] T b4a2db0198774bb2b4f3df769260e32f P 13d05ebdca8c4f489c597ce550dd773d -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 07:59:25.743430 8728 consensus_peers.cc:485] T 0f9e2549fe294a93b3b25081a15b2028 P 74077b61d6f54342bf9736b4a9f2eade -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 07:59:25.794924 6511 consensus_peers.cc:485] T af718dc43add4edfac703013655699f4 P 3b164cbbc00f4e8eb3254ddd349a3ff9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 07:59:25.835018 14202 raft_consensus.cc:482] T 2976d6353e8947b7b6c37f72efbeeac4 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83f20e84af4c40d09a4804b6b26e697f)
I20240501 07:59:25.835179 14202 raft_consensus.cc:504] T 2976d6353e8947b7b6c37f72efbeeac4 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: false } }
I20240501 07:59:25.835800 14202 leader_election.cc:288] T 2976d6353e8947b7b6c37f72efbeeac4 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607), 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149)
W20240501 07:59:25.836930 6511 leader_election.cc:334] T 2976d6353e8947b7b6c37f72efbeeac4 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111)
W20240501 07:59:25.837070 6511 leader_election.cc:334] T 2976d6353e8947b7b6c37f72efbeeac4 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111)
I20240501 07:59:25.837124 6511 leader_election.cc:302] T 2976d6353e8947b7b6c37f72efbeeac4 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b164cbbc00f4e8eb3254ddd349a3ff9; no voters: 83f20e84af4c40d09a4804b6b26e697f, a505e7cb133c4392bfda8a4a20874ae9
I20240501 07:59:25.837252 14202 raft_consensus.cc:2726] T 2976d6353e8947b7b6c37f72efbeeac4 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 07:59:26.027156 6511 consensus_peers.cc:462] T 0b24f67cf1a64debb23ee6618c820d33 P 3b164cbbc00f4e8eb3254ddd349a3ff9 -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111)
W20240501 07:59:26.070861 8859 consensus_peers.cc:485] T 12ce6a21d06c4ba889542758be9dc62d P 13d05ebdca8c4f489c597ce550dd773d -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 07:59:26.078733 8859 consensus_peers.cc:485] T 8a39f168c8f846cd9b8c4e2f2cf62fdd P 13d05ebdca8c4f489c597ce550dd773d -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 07:59:26.147686 8859 consensus_peers.cc:485] T c16b7962c18d4e02b850c7c7c678eae4 P 13d05ebdca8c4f489c597ce550dd773d -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 07:59:26.219023 6642 consensus_peers.cc:462] T a29720541b7b4a79aaa810592ef7c4b1 P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111)
W20240501 07:59:26.272630 6380 consensus_peers.cc:485] T 72aa4dc7664b48b7a73d34feba7d44d3 P 8315d22c56f7439a9e1fc82f153f018c -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 07:59:26.278251 6380 consensus_peers.cc:485] T f88fe033bce14bf0b1b63a52a7ba137e P 8315d22c56f7439a9e1fc82f153f018c -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 07:59:26.404352 6249 consensus_peers.cc:485] T b7d57b7fd5234f50afbc859bc997f3e1 P 8b8b3ad3aa8f4053b2167dd550f2086f -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 07:59:26.455322 8859 consensus_peers.cc:485] T 8f36eefcda3f475d9571e92fdeb52eb4 P 13d05ebdca8c4f489c597ce550dd773d -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 07:59:26.473649 14208 raft_consensus.cc:482] T a93ce5841c7f40a9ab23f88efe7b85bb P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83f20e84af4c40d09a4804b6b26e697f)
I20240501 07:59:26.473788 14208 raft_consensus.cc:504] T a93ce5841c7f40a9ab23f88efe7b85bb P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { replace: true } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { promote: true } }
I20240501 07:59:26.474223 14208 leader_election.cc:288] T a93ce5841c7f40a9ab23f88efe7b85bb P 74077b61d6f54342bf9736b4a9f2eade [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149), a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607)
W20240501 07:59:26.475291 8728 leader_election.cc:334] T a93ce5841c7f40a9ab23f88efe7b85bb P 74077b61d6f54342bf9736b4a9f2eade [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111)
W20240501 07:59:26.475392 8728 leader_election.cc:334] T a93ce5841c7f40a9ab23f88efe7b85bb P 74077b61d6f54342bf9736b4a9f2eade [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111)
I20240501 07:59:26.475443 8728 leader_election.cc:302] T a93ce5841c7f40a9ab23f88efe7b85bb P 74077b61d6f54342bf9736b4a9f2eade [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74077b61d6f54342bf9736b4a9f2eade; no voters: 83f20e84af4c40d09a4804b6b26e697f, a505e7cb133c4392bfda8a4a20874ae9
I20240501 07:59:26.475590 14208 raft_consensus.cc:2726] T a93ce5841c7f40a9ab23f88efe7b85bb P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 07:59:26.479557 8859 consensus_peers.cc:485] T fb6b46072a8d46e18b53b9f8de702859 P 13d05ebdca8c4f489c597ce550dd773d -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 07:59:26.495481 6380 consensus_peers.cc:485] T 4664b04b23294ad4b0a0e5c87fdca2ad P 8315d22c56f7439a9e1fc82f153f018c -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 07:59:26.563385 14208 raft_consensus.cc:482] T 4d4f47308d174a92bc73f23f2242be9a P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83f20e84af4c40d09a4804b6b26e697f)
I20240501 07:59:26.563517 14208 raft_consensus.cc:504] T 4d4f47308d174a92bc73f23f2242be9a P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { replace: true } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { promote: true } }
I20240501 07:59:26.563961 14208 leader_election.cc:288] T 4d4f47308d174a92bc73f23f2242be9a P 74077b61d6f54342bf9736b4a9f2eade [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149), a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607)
W20240501 07:59:26.565138 8728 leader_election.cc:334] T 4d4f47308d174a92bc73f23f2242be9a P 74077b61d6f54342bf9736b4a9f2eade [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111)
W20240501 07:59:26.565256 8728 leader_election.cc:334] T 4d4f47308d174a92bc73f23f2242be9a P 74077b61d6f54342bf9736b4a9f2eade [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111)
I20240501 07:59:26.565304 8728 leader_election.cc:302] T 4d4f47308d174a92bc73f23f2242be9a P 74077b61d6f54342bf9736b4a9f2eade [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74077b61d6f54342bf9736b4a9f2eade; no voters: 83f20e84af4c40d09a4804b6b26e697f, a505e7cb133c4392bfda8a4a20874ae9
I20240501 07:59:26.565439 14208 raft_consensus.cc:2726] T 4d4f47308d174a92bc73f23f2242be9a P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 07:59:26.626408 6249 consensus_peers.cc:485] T 5ee738571af740c1b494d4f17413d1d8 P 8b8b3ad3aa8f4053b2167dd550f2086f -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 07:59:26.632619 6511 consensus_peers.cc:485] T 6e2ac8c8857d45b88d3b51147d2609aa P 3b164cbbc00f4e8eb3254ddd349a3ff9 -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 07:59:26.635838 6511 consensus_peers.cc:462] T 0b24f67cf1a64debb23ee6618c820d33 P 3b164cbbc00f4e8eb3254ddd349a3ff9 -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111)
W20240501 07:59:26.658291 8859 consensus_peers.cc:485] T c197ba256dc74e6d92e34799c274b4e3 P 13d05ebdca8c4f489c597ce550dd773d -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 07:59:26.667446 6642 consensus_peers.cc:485] T 2560b1ec81ac4357b32c7a8ba9d72216 P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 07:59:26.669652 6642 consensus_peers.cc:485] T 3b9b71d44f754353b2321e8c36dd4dfd P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 07:59:26.690182 6380 consensus_peers.cc:485] T c3a9036ccfee45498bca9681518d3699 P 8315d22c56f7439a9e1fc82f153f018c -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 07:59:26.737198 6642 consensus_peers.cc:485] T 4bc915ffbbf345cbbe1195f8dd3ab874 P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 07:59:26.738224 6642 consensus_peers.cc:485] T 404bb81b9c404dd288d2f8160a4273da P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 07:59:26.738386 8859 consensus_peers.cc:485] T ff9dc384dbac47dc9795617c64688943 P 13d05ebdca8c4f489c597ce550dd773d -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 07:59:26.783540 6380 consensus_peers.cc:485] T 782e37a4ebb34a68b461a74115ae8703 P 8315d22c56f7439a9e1fc82f153f018c -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 07:59:26.803537 6380 consensus_peers.cc:485] T d726533fb72d4827991e12034b42634e P 8315d22c56f7439a9e1fc82f153f018c -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 07:59:26.810421 6642 consensus_peers.cc:462] T a29720541b7b4a79aaa810592ef7c4b1 P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111)
W20240501 07:59:26.851137 8859 consensus_peers.cc:485] T bab84387a40d40c3b8af7f09eb462886 P 13d05ebdca8c4f489c597ce550dd773d -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 07:59:26.866484 6511 consensus_peers.cc:485] T 114b6dbafa1c4c44ac42f1d2293e3125 P 3b164cbbc00f4e8eb3254ddd349a3ff9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 07:59:26.871780 6642 consensus_peers.cc:485] T a29720541b7b4a79aaa810592ef7c4b1 P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 07:59:26.901491 6249 consensus_peers.cc:485] T 0fc1ce00a2024e4a9c19abcc7d181efd P 8b8b3ad3aa8f4053b2167dd550f2086f -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 07:59:26.938999 6642 consensus_peers.cc:485] T 8454f24b0c28451b8ee51be39ca3e07f P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 07:59:26.954319 6249 consensus_peers.cc:485] T 6c3c0d6fe41f4ae196ae29beb9b68125 P 8b8b3ad3aa8f4053b2167dd550f2086f -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 07:59:26.987886 8859 consensus_peers.cc:485] T 239ee596e62d47169eeb6670dc62d59c P 13d05ebdca8c4f489c597ce550dd773d -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 07:59:27.021256 6380 consensus_peers.cc:485] T 782e37a4ebb34a68b461a74115ae8703 P 8315d22c56f7439a9e1fc82f153f018c -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 07:59:27.040742 6380 consensus_peers.cc:485] T bf806039d8c742d8a80ad994829bdf18 P 8315d22c56f7439a9e1fc82f153f018c -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 07:59:27.056627 8859 consensus_peers.cc:485] T e0583fa43994452ca7553cbf2309eba1 P 13d05ebdca8c4f489c597ce550dd773d -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 07:59:27.072845 6511 consensus_peers.cc:462] T 0b24f67cf1a64debb23ee6618c820d33 P 3b164cbbc00f4e8eb3254ddd349a3ff9 -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111)
I20240501 07:59:27.112591 14208 raft_consensus.cc:482] T 7dbbfc5d5848486f983cffdd2f36cc91 P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:27.112742 14208 raft_consensus.cc:504] T 7dbbfc5d5848486f983cffdd2f36cc91 P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } }
I20240501 07:59:27.113183 14208 leader_election.cc:288] T 7dbbfc5d5848486f983cffdd2f36cc91 P 74077b61d6f54342bf9736b4a9f2eade [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149), a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607)
W20240501 07:59:27.114372 8728 leader_election.cc:334] T 7dbbfc5d5848486f983cffdd2f36cc91 P 74077b61d6f54342bf9736b4a9f2eade [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111)
W20240501 07:59:27.114504 8728 leader_election.cc:334] T 7dbbfc5d5848486f983cffdd2f36cc91 P 74077b61d6f54342bf9736b4a9f2eade [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111)
I20240501 07:59:27.114565 8728 leader_election.cc:302] T 7dbbfc5d5848486f983cffdd2f36cc91 P 74077b61d6f54342bf9736b4a9f2eade [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74077b61d6f54342bf9736b4a9f2eade; no voters: 83f20e84af4c40d09a4804b6b26e697f, a505e7cb133c4392bfda8a4a20874ae9
I20240501 07:59:27.114725 14208 raft_consensus.cc:2726] T 7dbbfc5d5848486f983cffdd2f36cc91 P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 07:59:27.123461 8728 consensus_peers.cc:485] T ab1666f1e5204416adb435d9bbe80433 P 74077b61d6f54342bf9736b4a9f2eade -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 07:59:27.134024 6642 consensus_peers.cc:485] T 6daebcfc41754ec4aca939ca20588854 P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 07:59:27.150954 6511 consensus_peers.cc:485] T 3745acee2b5543b697a3c8f45d2618bb P 3b164cbbc00f4e8eb3254ddd349a3ff9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 07:59:27.153802 8859 consensus_peers.cc:485] T c197ba256dc74e6d92e34799c274b4e3 P 13d05ebdca8c4f489c597ce550dd773d -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 07:59:27.195592 6380 consensus_peers.cc:485] T cfcc814f4d2d488ba745ecaf0cc6d61a P 8315d22c56f7439a9e1fc82f153f018c -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 07:59:27.206709 6511 consensus_peers.cc:485] T d6c28cd91f624673b8e428f471a1dcce P 3b164cbbc00f4e8eb3254ddd349a3ff9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 07:59:27.270473 6380 consensus_peers.cc:485] T bf806039d8c742d8a80ad994829bdf18 P 8315d22c56f7439a9e1fc82f153f018c -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 07:59:27.297448 6642 consensus_peers.cc:462] T a29720541b7b4a79aaa810592ef7c4b1 P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111)
W20240501 07:59:27.353595 8728 consensus_peers.cc:485] T fcb3198c1fa5449781dc84756fe02bff P 74077b61d6f54342bf9736b4a9f2eade -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 07:59:27.374986 8859 consensus_peers.cc:485] T e0583fa43994452ca7553cbf2309eba1 P 13d05ebdca8c4f489c597ce550dd773d -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 07:59:27.383616 6249 consensus_peers.cc:485] T 2e540f389b2a4aef93a020f549d11b67 P 8b8b3ad3aa8f4053b2167dd550f2086f -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 07:59:27.390393 8728 consensus_peers.cc:485] T f3375300a7914fa599fe0d3a8f1f9c18 P 74077b61d6f54342bf9736b4a9f2eade -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 07:59:27.390995 14217 raft_consensus.cc:482] T c86c420310ec42ffa5b24304fba74267 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83f20e84af4c40d09a4804b6b26e697f)
I20240501 07:59:27.391122 14217 raft_consensus.cc:504] T c86c420310ec42ffa5b24304fba74267 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { replace: true } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: true } }
I20240501 07:59:27.391536 14217 leader_election.cc:288] T c86c420310ec42ffa5b24304fba74267 P 8315d22c56f7439a9e1fc82f153f018c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149), a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607)
W20240501 07:59:27.392728 6380 leader_election.cc:334] T c86c420310ec42ffa5b24304fba74267 P 8315d22c56f7439a9e1fc82f153f018c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111)
W20240501 07:59:27.392848 6380 leader_election.cc:334] T c86c420310ec42ffa5b24304fba74267 P 8315d22c56f7439a9e1fc82f153f018c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111)
I20240501 07:59:27.392897 6380 leader_election.cc:302] T c86c420310ec42ffa5b24304fba74267 P 8315d22c56f7439a9e1fc82f153f018c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8315d22c56f7439a9e1fc82f153f018c; no voters: 83f20e84af4c40d09a4804b6b26e697f, a505e7cb133c4392bfda8a4a20874ae9
I20240501 07:59:27.393051 14217 raft_consensus.cc:2726] T c86c420310ec42ffa5b24304fba74267 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 07:59:27.405802 6380 consensus_peers.cc:485] T 9461ad0f65e1489ca102f2558194ce8f P 8315d22c56f7439a9e1fc82f153f018c -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 07:59:27.449430 8728 consensus_peers.cc:485] T c5a876dfb598428ab350593129505504 P 74077b61d6f54342bf9736b4a9f2eade -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 07:59:27.469414 6511 consensus_peers.cc:462] T 0b24f67cf1a64debb23ee6618c820d33 P 3b164cbbc00f4e8eb3254ddd349a3ff9 -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111)
W20240501 07:59:27.505352 6511 consensus_peers.cc:485] T d3f2250a62c6429e886753aaffcbe883 P 3b164cbbc00f4e8eb3254ddd349a3ff9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 07:59:27.543375 8859 consensus_peers.cc:485] T f0c1c05110894e39b41e6b5c5894b8d8 P 13d05ebdca8c4f489c597ce550dd773d -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 07:59:27.574524 6249 consensus_peers.cc:485] T f68dbc1f65eb4f1b91a62e2de82a7676 P 8b8b3ad3aa8f4053b2167dd550f2086f -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 07:59:27.576546 6511 consensus_peers.cc:485] T d3f2250a62c6429e886753aaffcbe883 P 3b164cbbc00f4e8eb3254ddd349a3ff9 -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 07:59:27.668170 6642 consensus_peers.cc:485] T 28eaf22342ff4b5394f958d0b659273c P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 07:59:27.678313 6380 consensus_peers.cc:485] T a756205204d146ce82a2799e30582f57 P 8315d22c56f7439a9e1fc82f153f018c -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 07:59:27.679584 6642 consensus_peers.cc:462] T a29720541b7b4a79aaa810592ef7c4b1 P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111)
W20240501 07:59:27.689492 8859 consensus_peers.cc:485] T ad3e528eaff349a4b3b03037652b1505 P 13d05ebdca8c4f489c597ce550dd773d -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 07:59:27.743005 8728 consensus_peers.cc:485] T 6dc929e2a6ae45bd8b1f297583636bb6 P 74077b61d6f54342bf9736b4a9f2eade -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 07:59:27.782116 8859 consensus_peers.cc:485] T 0864c290fafa488abefa93d41c0a2efe P 13d05ebdca8c4f489c597ce550dd773d -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 07:59:27.885301 8728 consensus_peers.cc:485] T 36aaf192042f4320b2526322424dc640 P 74077b61d6f54342bf9736b4a9f2eade -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 07:59:27.967216 6511 consensus_peers.cc:462] T 0b24f67cf1a64debb23ee6618c820d33 P 3b164cbbc00f4e8eb3254ddd349a3ff9 -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111)
W20240501 07:59:27.983130 6249 consensus_peers.cc:485] T 1c32cd00bc344ae1b161d3f9b43b53d3 P 8b8b3ad3aa8f4053b2167dd550f2086f -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 07:59:27.990888 14195 raft_consensus.cc:482] T 1de93edea7714491ae339db8cdfec090 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a505e7cb133c4392bfda8a4a20874ae9)
I20240501 07:59:27.991035 14195 raft_consensus.cc:504] T 1de93edea7714491ae339db8cdfec090 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } }
I20240501 07:59:27.991444 14195 leader_election.cc:288] T 1de93edea7714491ae339db8cdfec090 P 8315d22c56f7439a9e1fc82f153f018c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149), a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607)
W20240501 07:59:27.992554 6380 leader_election.cc:334] T 1de93edea7714491ae339db8cdfec090 P 8315d22c56f7439a9e1fc82f153f018c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111)
W20240501 07:59:27.992684 6380 leader_election.cc:334] T 1de93edea7714491ae339db8cdfec090 P 8315d22c56f7439a9e1fc82f153f018c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111)
I20240501 07:59:27.992740 6380 leader_election.cc:302] T 1de93edea7714491ae339db8cdfec090 P 8315d22c56f7439a9e1fc82f153f018c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8315d22c56f7439a9e1fc82f153f018c; no voters: 83f20e84af4c40d09a4804b6b26e697f, a505e7cb133c4392bfda8a4a20874ae9
I20240501 07:59:27.992902 14195 raft_consensus.cc:2726] T 1de93edea7714491ae339db8cdfec090 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 07:59:28.020870 8728 consensus_peers.cc:485] T e5bd3a8f7a7640c09774f781009eb19c P 74077b61d6f54342bf9736b4a9f2eade -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 07:59:28.074718 6158 catalog_manager.cc:4573] TS 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): CreateTablet RPC failed for tablet 7dbbfc5d5848486f983cffdd2f36cc91: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111)
W20240501 07:59:28.097550 8859 consensus_peers.cc:485] T b4a2db0198774bb2b4f3df769260e32f P 13d05ebdca8c4f489c597ce550dd773d -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 07:59:28.135432 6642 consensus_peers.cc:462] T a29720541b7b4a79aaa810592ef7c4b1 P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111)
W20240501 07:59:28.287817 8859 consensus_peers.cc:485] T f788de4df5454d00ac9bb883f0abc9bd P 13d05ebdca8c4f489c597ce550dd773d -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 07:59:28.341161 6511 consensus_peers.cc:485] T af718dc43add4edfac703013655699f4 P 3b164cbbc00f4e8eb3254ddd349a3ff9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 07:59:28.418514 8859 consensus_peers.cc:485] T 8a39f168c8f846cd9b8c4e2f2cf62fdd P 13d05ebdca8c4f489c597ce550dd773d -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 07:59:28.427412 8859 consensus_peers.cc:485] T c16b7962c18d4e02b850c7c7c678eae4 P 13d05ebdca8c4f489c597ce550dd773d -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 07:59:28.469605 6511 consensus_peers.cc:462] T 0b24f67cf1a64debb23ee6618c820d33 P 3b164cbbc00f4e8eb3254ddd349a3ff9 -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111)
W20240501 07:59:28.527129 8859 consensus_peers.cc:485] T 12ce6a21d06c4ba889542758be9dc62d P 13d05ebdca8c4f489c597ce550dd773d -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 07:59:28.527348 14208 raft_consensus.cc:482] T a93ce5841c7f40a9ab23f88efe7b85bb P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83f20e84af4c40d09a4804b6b26e697f)
I20240501 07:59:28.527488 14208 raft_consensus.cc:504] T a93ce5841c7f40a9ab23f88efe7b85bb P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { replace: true } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { promote: true } }
I20240501 07:59:28.527877 14208 leader_election.cc:288] T a93ce5841c7f40a9ab23f88efe7b85bb P 74077b61d6f54342bf9736b4a9f2eade [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149), a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607)
W20240501 07:59:28.528856 8728 leader_election.cc:334] T a93ce5841c7f40a9ab23f88efe7b85bb P 74077b61d6f54342bf9736b4a9f2eade [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111)
W20240501 07:59:28.529003 8728 leader_election.cc:334] T a93ce5841c7f40a9ab23f88efe7b85bb P 74077b61d6f54342bf9736b4a9f2eade [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111)
I20240501 07:59:28.529057 8728 leader_election.cc:302] T a93ce5841c7f40a9ab23f88efe7b85bb P 74077b61d6f54342bf9736b4a9f2eade [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74077b61d6f54342bf9736b4a9f2eade; no voters: 83f20e84af4c40d09a4804b6b26e697f, a505e7cb133c4392bfda8a4a20874ae9
I20240501 07:59:28.529187 14208 raft_consensus.cc:2726] T a93ce5841c7f40a9ab23f88efe7b85bb P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 07:59:28.617168 8728 consensus_peers.cc:485] T 0f9e2549fe294a93b3b25081a15b2028 P 74077b61d6f54342bf9736b4a9f2eade -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 07:59:28.656720 6642 consensus_peers.cc:462] T a29720541b7b4a79aaa810592ef7c4b1 P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111)
W20240501 07:59:28.687516 6380 consensus_peers.cc:485] T f88fe033bce14bf0b1b63a52a7ba137e P 8315d22c56f7439a9e1fc82f153f018c -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 07:59:28.885974 6249 consensus_peers.cc:485] T b7d57b7fd5234f50afbc859bc997f3e1 P 8b8b3ad3aa8f4053b2167dd550f2086f -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 07:59:28.960150 6380 consensus_peers.cc:485] T 72aa4dc7664b48b7a73d34feba7d44d3 P 8315d22c56f7439a9e1fc82f153f018c -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 07:59:28.962262 6511 consensus_peers.cc:462] T 0b24f67cf1a64debb23ee6618c820d33 P 3b164cbbc00f4e8eb3254ddd349a3ff9 -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111)
I20240501 07:59:28.967473 14202 raft_consensus.cc:482] T e81b23661a1d475887c63f79d8f90ab6 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:28.967600 14202 raft_consensus.cc:504] T e81b23661a1d475887c63f79d8f90ab6 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } }
I20240501 07:59:28.967993 14202 leader_election.cc:288] T e81b23661a1d475887c63f79d8f90ab6 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149), a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607)
W20240501 07:59:28.968935 6511 leader_election.cc:334] T e81b23661a1d475887c63f79d8f90ab6 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111)
W20240501 07:59:28.969062 6511 leader_election.cc:334] T e81b23661a1d475887c63f79d8f90ab6 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111)
I20240501 07:59:28.969097 6511 leader_election.cc:302] T e81b23661a1d475887c63f79d8f90ab6 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b164cbbc00f4e8eb3254ddd349a3ff9; no voters: 83f20e84af4c40d09a4804b6b26e697f, a505e7cb133c4392bfda8a4a20874ae9
I20240501 07:59:28.969204 14202 raft_consensus.cc:2726] T e81b23661a1d475887c63f79d8f90ab6 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240501 07:59:28.977888 6642 consensus_peers.cc:485] T 404bb81b9c404dd288d2f8160a4273da P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 07:59:28.990085 8859 consensus_peers.cc:485] T ff9dc384dbac47dc9795617c64688943 P 13d05ebdca8c4f489c597ce550dd773d -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 07:59:29.057508 8859 consensus_peers.cc:485] T fb6b46072a8d46e18b53b9f8de702859 P 13d05ebdca8c4f489c597ce550dd773d -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 07:59:29.066365 6380 consensus_peers.cc:485] T 4664b04b23294ad4b0a0e5c87fdca2ad P 8315d22c56f7439a9e1fc82f153f018c -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 07:59:29.072328 6249 consensus_peers.cc:485] T 0fc1ce00a2024e4a9c19abcc7d181efd P 8b8b3ad3aa8f4053b2167dd550f2086f -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 07:59:29.080897 6642 consensus_peers.cc:485] T 2560b1ec81ac4357b32c7a8ba9d72216 P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 07:59:29.095676 6249 consensus_peers.cc:485] T 5ee738571af740c1b494d4f17413d1d8 P 8b8b3ad3aa8f4053b2167dd550f2086f -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 07:59:29.099220 6511 consensus_peers.cc:485] T 114b6dbafa1c4c44ac42f1d2293e3125 P 3b164cbbc00f4e8eb3254ddd349a3ff9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 07:59:29.133478 8859 consensus_peers.cc:485] T 8f36eefcda3f475d9571e92fdeb52eb4 P 13d05ebdca8c4f489c597ce550dd773d -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 07:59:29.153362 6511 consensus_peers.cc:485] T 6e2ac8c8857d45b88d3b51147d2609aa P 3b164cbbc00f4e8eb3254ddd349a3ff9 -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 07:59:29.198634 6642 consensus_peers.cc:485] T 4bc915ffbbf345cbbe1195f8dd3ab874 P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 07:59:29.207485 6642 consensus_peers.cc:462] T a29720541b7b4a79aaa810592ef7c4b1 P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111)
W20240501 07:59:29.215690 6380 consensus_peers.cc:485] T c3a9036ccfee45498bca9681518d3699 P 8315d22c56f7439a9e1fc82f153f018c -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 07:59:29.255949 6642 consensus_peers.cc:485] T 3b9b71d44f754353b2321e8c36dd4dfd P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 07:59:29.292830 8859 consensus_peers.cc:485] T 239ee596e62d47169eeb6670dc62d59c P 13d05ebdca8c4f489c597ce550dd773d -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 07:59:29.311651 8859 consensus_peers.cc:485] T c197ba256dc74e6d92e34799c274b4e3 P 13d05ebdca8c4f489c597ce550dd773d -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 07:59:29.387229 8859 consensus_peers.cc:485] T e0583fa43994452ca7553cbf2309eba1 P 13d05ebdca8c4f489c597ce550dd773d -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 07:59:29.434927 6380 consensus_peers.cc:485] T 782e37a4ebb34a68b461a74115ae8703 P 8315d22c56f7439a9e1fc82f153f018c -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 07:59:29.437553 6511 consensus_peers.cc:462] T 0b24f67cf1a64debb23ee6618c820d33 P 3b164cbbc00f4e8eb3254ddd349a3ff9 -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111)
W20240501 07:59:29.452003 8859 consensus_peers.cc:485] T bab84387a40d40c3b8af7f09eb462886 P 13d05ebdca8c4f489c597ce550dd773d -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 07:59:29.472541 6158 catalog_manager.cc:4573] TS a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): CreateTablet RPC failed for tablet e81b23661a1d475887c63f79d8f90ab6: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111)
W20240501 07:59:29.522666 6249 consensus_peers.cc:485] T 6c3c0d6fe41f4ae196ae29beb9b68125 P 8b8b3ad3aa8f4053b2167dd550f2086f -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 07:59:29.522789 6380 consensus_peers.cc:485] T d726533fb72d4827991e12034b42634e P 8315d22c56f7439a9e1fc82f153f018c -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 07:59:29.524919 6511 consensus_peers.cc:485] T 3745acee2b5543b697a3c8f45d2618bb P 3b164cbbc00f4e8eb3254ddd349a3ff9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 07:59:29.529958 8728 consensus_peers.cc:485] T c5a876dfb598428ab350593129505504 P 74077b61d6f54342bf9736b4a9f2eade -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 07:59:29.549976 6511 consensus_peers.cc:485] T d6c28cd91f624673b8e428f471a1dcce P 3b164cbbc00f4e8eb3254ddd349a3ff9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 07:59:29.591176 6642 consensus_peers.cc:485] T a29720541b7b4a79aaa810592ef7c4b1 P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 07:59:29.591905 6249 consensus_peers.cc:485] T 2e540f389b2a4aef93a020f549d11b67 P 8b8b3ad3aa8f4053b2167dd550f2086f -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 07:59:29.600617 6642 consensus_peers.cc:462] T a29720541b7b4a79aaa810592ef7c4b1 P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111)
W20240501 07:59:29.624400 6642 consensus_peers.cc:485] T 8454f24b0c28451b8ee51be39ca3e07f P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 07:59:29.632773 6380 consensus_peers.cc:485] T bf806039d8c742d8a80ad994829bdf18 P 8315d22c56f7439a9e1fc82f153f018c -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 07:59:29.634994 8728 consensus_peers.cc:485] T f3375300a7914fa599fe0d3a8f1f9c18 P 74077b61d6f54342bf9736b4a9f2eade -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 07:59:29.651401 8728 consensus_peers.cc:485] T ab1666f1e5204416adb435d9bbe80433 P 74077b61d6f54342bf9736b4a9f2eade -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 07:59:29.655895 8859 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111) [suppressed 169 similar messages]
W20240501 07:59:29.680894 8859 consensus_peers.cc:485] T c197ba256dc74e6d92e34799c274b4e3 P 13d05ebdca8c4f489c597ce550dd773d -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 07:59:29.698084 6380 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111) [suppressed 127 similar messages]
W20240501 07:59:29.698683 6380 consensus_peers.cc:485] T 782e37a4ebb34a68b461a74115ae8703 P 8315d22c56f7439a9e1fc82f153f018c -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 07:59:29.708252 6380 consensus_peers.cc:485] T bf806039d8c742d8a80ad994829bdf18 P 8315d22c56f7439a9e1fc82f153f018c -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 07:59:29.714442 6511 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111) [suppressed 86 similar messages]
W20240501 07:59:29.716439 6380 consensus_peers.cc:485] T cfcc814f4d2d488ba745ecaf0cc6d61a P 8315d22c56f7439a9e1fc82f153f018c -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 07:59:29.750209 6642 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111) [suppressed 92 similar messages]
W20240501 07:59:29.801797 6249 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111) [suppressed 72 similar messages]
W20240501 07:59:29.806804 8728 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111) [suppressed 92 similar messages]
W20240501 07:59:29.807379 8728 consensus_peers.cc:485] T fcb3198c1fa5449781dc84756fe02bff P 74077b61d6f54342bf9736b4a9f2eade -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 07:59:29.812382 6380 consensus_peers.cc:485] T 9461ad0f65e1489ca102f2558194ce8f P 8315d22c56f7439a9e1fc82f153f018c -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 07:59:29.838222 6249 consensus_peers.cc:485] T f68dbc1f65eb4f1b91a62e2de82a7676 P 8b8b3ad3aa8f4053b2167dd550f2086f -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 07:59:29.862260 8859 consensus_peers.cc:485] T e0583fa43994452ca7553cbf2309eba1 P 13d05ebdca8c4f489c597ce550dd773d -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 07:59:29.868536 6642 consensus_peers.cc:485] T 6daebcfc41754ec4aca939ca20588854 P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 07:59:29.953832 6511 consensus_peers.cc:462] T 0b24f67cf1a64debb23ee6618c820d33 P 3b164cbbc00f4e8eb3254ddd349a3ff9 -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111)
W20240501 07:59:29.959832 6511 consensus_peers.cc:485] T d3f2250a62c6429e886753aaffcbe883 P 3b164cbbc00f4e8eb3254ddd349a3ff9 -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 07:59:30.029177 6380 consensus_peers.cc:485] T a756205204d146ce82a2799e30582f57 P 8315d22c56f7439a9e1fc82f153f018c -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 07:59:30.030390 8859 consensus_peers.cc:485] T 0864c290fafa488abefa93d41c0a2efe P 13d05ebdca8c4f489c597ce550dd773d -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 07:59:30.133998 6642 consensus_peers.cc:462] T a29720541b7b4a79aaa810592ef7c4b1 P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111)
W20240501 07:59:30.144731 8859 consensus_peers.cc:485] T f0c1c05110894e39b41e6b5c5894b8d8 P 13d05ebdca8c4f489c597ce550dd773d -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 07:59:30.192627 8859 consensus_peers.cc:485] T ad3e528eaff349a4b3b03037652b1505 P 13d05ebdca8c4f489c597ce550dd773d -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 07:59:30.197180 6642 consensus_peers.cc:485] T 28eaf22342ff4b5394f958d0b659273c P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 07:59:30.230394 14230 raft_consensus.cc:482] T 2976d6353e8947b7b6c37f72efbeeac4 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83f20e84af4c40d09a4804b6b26e697f)
I20240501 07:59:30.230540 14230 raft_consensus.cc:504] T 2976d6353e8947b7b6c37f72efbeeac4 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: false } }
I20240501 07:59:30.231063 14230 leader_election.cc:288] T 2976d6353e8947b7b6c37f72efbeeac4 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607), 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149)
W20240501 07:59:30.232385 6511 leader_election.cc:334] T 2976d6353e8947b7b6c37f72efbeeac4 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111)
W20240501 07:59:30.232487 6511 leader_election.cc:334] T 2976d6353e8947b7b6c37f72efbeeac4 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111)
I20240501 07:59:30.232542 6511 leader_election.cc:302] T 2976d6353e8947b7b6c37f72efbeeac4 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b164cbbc00f4e8eb3254ddd349a3ff9; no voters: 83f20e84af4c40d09a4804b6b26e697f, a505e7cb133c4392bfda8a4a20874ae9
I20240501 07:59:30.232698 14230 raft_consensus.cc:2726] T 2976d6353e8947b7b6c37f72efbeeac4 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 07:59:30.257355 6511 consensus_peers.cc:485] T d3f2250a62c6429e886753aaffcbe883 P 3b164cbbc00f4e8eb3254ddd349a3ff9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 07:59:30.275581 6249 consensus_peers.cc:485] T 1c32cd00bc344ae1b161d3f9b43b53d3 P 8b8b3ad3aa8f4053b2167dd550f2086f -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 07:59:30.286545 8728 consensus_peers.cc:485] T 36aaf192042f4320b2526322424dc640 P 74077b61d6f54342bf9736b4a9f2eade -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 07:59:30.413448 8728 consensus_peers.cc:485] T 6dc929e2a6ae45bd8b1f297583636bb6 P 74077b61d6f54342bf9736b4a9f2eade -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 07:59:30.443861 6511 consensus_peers.cc:462] T 0b24f67cf1a64debb23ee6618c820d33 P 3b164cbbc00f4e8eb3254ddd349a3ff9 -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111)
I20240501 07:59:30.568995 14231 raft_consensus.cc:482] T 13daf8496cd64d7d959ce5ad666b9be8 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a505e7cb133c4392bfda8a4a20874ae9)
I20240501 07:59:30.569193 14231 raft_consensus.cc:504] T 13daf8496cd64d7d959ce5ad666b9be8 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } }
I20240501 07:59:30.569626 14231 leader_election.cc:288] T 13daf8496cd64d7d959ce5ad666b9be8 P 8315d22c56f7439a9e1fc82f153f018c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149), a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607)
W20240501 07:59:30.570514 6380 leader_election.cc:334] T 13daf8496cd64d7d959ce5ad666b9be8 P 8315d22c56f7439a9e1fc82f153f018c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111)
W20240501 07:59:30.570638 6380 leader_election.cc:334] T 13daf8496cd64d7d959ce5ad666b9be8 P 8315d22c56f7439a9e1fc82f153f018c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111)
I20240501 07:59:30.570675 6380 leader_election.cc:302] T 13daf8496cd64d7d959ce5ad666b9be8 P 8315d22c56f7439a9e1fc82f153f018c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8315d22c56f7439a9e1fc82f153f018c; no voters: 83f20e84af4c40d09a4804b6b26e697f, a505e7cb133c4392bfda8a4a20874ae9
I20240501 07:59:30.570789 14231 raft_consensus.cc:2726] T 13daf8496cd64d7d959ce5ad666b9be8 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 07:59:30.599218 8859 consensus_peers.cc:485] T b4a2db0198774bb2b4f3df769260e32f P 13d05ebdca8c4f489c597ce550dd773d -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 07:59:30.686875 8728 consensus_peers.cc:485] T e5bd3a8f7a7640c09774f781009eb19c P 74077b61d6f54342bf9736b4a9f2eade -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 07:59:30.688645 6642 consensus_peers.cc:462] T a29720541b7b4a79aaa810592ef7c4b1 P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111)
W20240501 07:59:30.690914 8859 consensus_peers.cc:485] T c16b7962c18d4e02b850c7c7c678eae4 P 13d05ebdca8c4f489c597ce550dd773d -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 07:59:30.722059 8859 consensus_peers.cc:485] T f788de4df5454d00ac9bb883f0abc9bd P 13d05ebdca8c4f489c597ce550dd773d -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 07:59:30.838601 14230 raft_consensus.cc:482] T e81b23661a1d475887c63f79d8f90ab6 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:30.838737 14230 raft_consensus.cc:504] T e81b23661a1d475887c63f79d8f90ab6 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } }
I20240501 07:59:30.839121 14230 leader_election.cc:288] T e81b23661a1d475887c63f79d8f90ab6 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149), a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607)
W20240501 07:59:30.840328 6511 leader_election.cc:334] T e81b23661a1d475887c63f79d8f90ab6 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111)
W20240501 07:59:30.840453 6511 leader_election.cc:334] T e81b23661a1d475887c63f79d8f90ab6 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111)
I20240501 07:59:30.840524 6511 leader_election.cc:302] T e81b23661a1d475887c63f79d8f90ab6 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b164cbbc00f4e8eb3254ddd349a3ff9; no voters: 83f20e84af4c40d09a4804b6b26e697f, a505e7cb133c4392bfda8a4a20874ae9
I20240501 07:59:30.840677 14230 raft_consensus.cc:2726] T e81b23661a1d475887c63f79d8f90ab6 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240501 07:59:30.852142 8859 consensus_peers.cc:485] T 8a39f168c8f846cd9b8c4e2f2cf62fdd P 13d05ebdca8c4f489c597ce550dd773d -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 07:59:30.927026 6511 consensus_peers.cc:462] T 0b24f67cf1a64debb23ee6618c820d33 P 3b164cbbc00f4e8eb3254ddd349a3ff9 -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111)
W20240501 07:59:31.054579 6511 consensus_peers.cc:485] T af718dc43add4edfac703013655699f4 P 3b164cbbc00f4e8eb3254ddd349a3ff9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 07:59:31.123337 8728 consensus_peers.cc:485] T 0f9e2549fe294a93b3b25081a15b2028 P 74077b61d6f54342bf9736b4a9f2eade -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 07:59:31.137282 8859 consensus_peers.cc:485] T 12ce6a21d06c4ba889542758be9dc62d P 13d05ebdca8c4f489c597ce550dd773d -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 07:59:31.171200 6380 consensus_peers.cc:485] T 72aa4dc7664b48b7a73d34feba7d44d3 P 8315d22c56f7439a9e1fc82f153f018c -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 07:59:31.258349 6642 consensus_peers.cc:485] T 404bb81b9c404dd288d2f8160a4273da P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 07:59:31.288414 6642 consensus_peers.cc:462] T a29720541b7b4a79aaa810592ef7c4b1 P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111)
W20240501 07:59:31.328509 8859 consensus_peers.cc:485] T ff9dc384dbac47dc9795617c64688943 P 13d05ebdca8c4f489c597ce550dd773d -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 07:59:31.348044 6380 consensus_peers.cc:485] T f88fe033bce14bf0b1b63a52a7ba137e P 8315d22c56f7439a9e1fc82f153f018c -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 07:59:31.370513 6511 consensus_peers.cc:462] T 0b24f67cf1a64debb23ee6618c820d33 P 3b164cbbc00f4e8eb3254ddd349a3ff9 -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111)
W20240501 07:59:31.399796 6249 consensus_peers.cc:485] T b7d57b7fd5234f50afbc859bc997f3e1 P 8b8b3ad3aa8f4053b2167dd550f2086f -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 07:59:31.430280 6249 consensus_peers.cc:485] T 5ee738571af740c1b494d4f17413d1d8 P 8b8b3ad3aa8f4053b2167dd550f2086f -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 07:59:31.478447 8859 consensus_peers.cc:485] T bab84387a40d40c3b8af7f09eb462886 P 13d05ebdca8c4f489c597ce550dd773d -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 07:59:31.489722 6511 consensus_peers.cc:485] T 114b6dbafa1c4c44ac42f1d2293e3125 P 3b164cbbc00f4e8eb3254ddd349a3ff9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 07:59:31.491276 6511 consensus_peers.cc:485] T 6e2ac8c8857d45b88d3b51147d2609aa P 3b164cbbc00f4e8eb3254ddd349a3ff9 -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 07:59:31.528668 6642 consensus_peers.cc:485] T 3b9b71d44f754353b2321e8c36dd4dfd P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 07:59:31.532605 6249 consensus_peers.cc:485] T 0fc1ce00a2024e4a9c19abcc7d181efd P 8b8b3ad3aa8f4053b2167dd550f2086f -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 07:59:31.549665 8859 consensus_peers.cc:485] T 8f36eefcda3f475d9571e92fdeb52eb4 P 13d05ebdca8c4f489c597ce550dd773d -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 07:59:31.608002 14244 raft_consensus.cc:482] T 4d4f47308d174a92bc73f23f2242be9a P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83f20e84af4c40d09a4804b6b26e697f)
I20240501 07:59:31.608141 14244 raft_consensus.cc:504] T 4d4f47308d174a92bc73f23f2242be9a P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { replace: true } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { promote: true } }
I20240501 07:59:31.608552 14244 leader_election.cc:288] T 4d4f47308d174a92bc73f23f2242be9a P 74077b61d6f54342bf9736b4a9f2eade [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149), a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607)
W20240501 07:59:31.609755 8728 leader_election.cc:334] T 4d4f47308d174a92bc73f23f2242be9a P 74077b61d6f54342bf9736b4a9f2eade [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111)
W20240501 07:59:31.609894 8728 leader_election.cc:334] T 4d4f47308d174a92bc73f23f2242be9a P 74077b61d6f54342bf9736b4a9f2eade [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111)
I20240501 07:59:31.609941 8728 leader_election.cc:302] T 4d4f47308d174a92bc73f23f2242be9a P 74077b61d6f54342bf9736b4a9f2eade [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74077b61d6f54342bf9736b4a9f2eade; no voters: 83f20e84af4c40d09a4804b6b26e697f, a505e7cb133c4392bfda8a4a20874ae9
I20240501 07:59:31.610085 14244 raft_consensus.cc:2726] T 4d4f47308d174a92bc73f23f2242be9a P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 07:59:31.652315 6642 consensus_peers.cc:485] T 4bc915ffbbf345cbbe1195f8dd3ab874 P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 07:59:31.677335 6249 consensus_peers.cc:485] T 6c3c0d6fe41f4ae196ae29beb9b68125 P 8b8b3ad3aa8f4053b2167dd550f2086f -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 07:59:31.691365 6380 consensus_peers.cc:485] T 4664b04b23294ad4b0a0e5c87fdca2ad P 8315d22c56f7439a9e1fc82f153f018c -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 07:59:31.723296 6642 consensus_peers.cc:485] T 2560b1ec81ac4357b32c7a8ba9d72216 P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 07:59:31.739996 8859 consensus_peers.cc:485] T 239ee596e62d47169eeb6670dc62d59c P 13d05ebdca8c4f489c597ce550dd773d -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 07:59:31.750250 8859 consensus_peers.cc:485] T fb6b46072a8d46e18b53b9f8de702859 P 13d05ebdca8c4f489c597ce550dd773d -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 07:59:31.755870 6642 consensus_peers.cc:462] T a29720541b7b4a79aaa810592ef7c4b1 P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111)
W20240501 07:59:31.765235 8859 consensus_peers.cc:485] T c197ba256dc74e6d92e34799c274b4e3 P 13d05ebdca8c4f489c597ce550dd773d -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 07:59:31.819334 6380 consensus_peers.cc:485] T c3a9036ccfee45498bca9681518d3699 P 8315d22c56f7439a9e1fc82f153f018c -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 07:59:31.861631 6511 consensus_peers.cc:462] T 0b24f67cf1a64debb23ee6618c820d33 P 3b164cbbc00f4e8eb3254ddd349a3ff9 -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111)
I20240501 07:59:31.887213 14242 raft_consensus.cc:482] T 1de93edea7714491ae339db8cdfec090 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a505e7cb133c4392bfda8a4a20874ae9)
I20240501 07:59:31.887343 14242 raft_consensus.cc:504] T 1de93edea7714491ae339db8cdfec090 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } }
I20240501 07:59:31.887732 14242 leader_election.cc:288] T 1de93edea7714491ae339db8cdfec090 P 8315d22c56f7439a9e1fc82f153f018c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149), a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607)
W20240501 07:59:31.888784 6380 leader_election.cc:334] T 1de93edea7714491ae339db8cdfec090 P 8315d22c56f7439a9e1fc82f153f018c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111)
W20240501 07:59:31.888906 6380 leader_election.cc:334] T 1de93edea7714491ae339db8cdfec090 P 8315d22c56f7439a9e1fc82f153f018c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111)
I20240501 07:59:31.888984 6380 leader_election.cc:302] T 1de93edea7714491ae339db8cdfec090 P 8315d22c56f7439a9e1fc82f153f018c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8315d22c56f7439a9e1fc82f153f018c; no voters: 83f20e84af4c40d09a4804b6b26e697f, a505e7cb133c4392bfda8a4a20874ae9
I20240501 07:59:31.889155 14242 raft_consensus.cc:2726] T 1de93edea7714491ae339db8cdfec090 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 07:59:31.904187 6380 consensus_peers.cc:485] T bf806039d8c742d8a80ad994829bdf18 P 8315d22c56f7439a9e1fc82f153f018c -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 07:59:31.910487 14244 raft_consensus.cc:482] T a93ce5841c7f40a9ab23f88efe7b85bb P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83f20e84af4c40d09a4804b6b26e697f)
I20240501 07:59:31.910619 14244 raft_consensus.cc:504] T a93ce5841c7f40a9ab23f88efe7b85bb P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { replace: true } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { promote: true } }
I20240501 07:59:31.911001 14244 leader_election.cc:288] T a93ce5841c7f40a9ab23f88efe7b85bb P 74077b61d6f54342bf9736b4a9f2eade [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149), a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607)
W20240501 07:59:31.912014 8728 leader_election.cc:334] T a93ce5841c7f40a9ab23f88efe7b85bb P 74077b61d6f54342bf9736b4a9f2eade [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111)
W20240501 07:59:31.912129 8728 leader_election.cc:334] T a93ce5841c7f40a9ab23f88efe7b85bb P 74077b61d6f54342bf9736b4a9f2eade [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111)
I20240501 07:59:31.912179 8728 leader_election.cc:302] T a93ce5841c7f40a9ab23f88efe7b85bb P 74077b61d6f54342bf9736b4a9f2eade [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74077b61d6f54342bf9736b4a9f2eade; no voters: 83f20e84af4c40d09a4804b6b26e697f, a505e7cb133c4392bfda8a4a20874ae9
I20240501 07:59:31.912317 14244 raft_consensus.cc:2726] T a93ce5841c7f40a9ab23f88efe7b85bb P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 07:59:31.940260 8859 consensus_peers.cc:485] T e0583fa43994452ca7553cbf2309eba1 P 13d05ebdca8c4f489c597ce550dd773d -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 07:59:31.961192 6511 consensus_peers.cc:485] T 3745acee2b5543b697a3c8f45d2618bb P 3b164cbbc00f4e8eb3254ddd349a3ff9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 07:59:31.974939 8859 consensus_peers.cc:485] T c197ba256dc74e6d92e34799c274b4e3 P 13d05ebdca8c4f489c597ce550dd773d -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 07:59:32.004666 8728 consensus_peers.cc:485] T c5a876dfb598428ab350593129505504 P 74077b61d6f54342bf9736b4a9f2eade -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 07:59:32.037883 8728 consensus_peers.cc:485] T fcb3198c1fa5449781dc84756fe02bff P 74077b61d6f54342bf9736b4a9f2eade -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 07:59:32.038983 6380 consensus_peers.cc:485] T d726533fb72d4827991e12034b42634e P 8315d22c56f7439a9e1fc82f153f018c -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 07:59:32.093477 6511 consensus_peers.cc:485] T d6c28cd91f624673b8e428f471a1dcce P 3b164cbbc00f4e8eb3254ddd349a3ff9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 07:59:32.096586 6642 consensus_peers.cc:485] T a29720541b7b4a79aaa810592ef7c4b1 P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 07:59:32.101918 8728 consensus_peers.cc:485] T ab1666f1e5204416adb435d9bbe80433 P 74077b61d6f54342bf9736b4a9f2eade -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 07:59:32.102017 8728 consensus_peers.cc:485] T f3375300a7914fa599fe0d3a8f1f9c18 P 74077b61d6f54342bf9736b4a9f2eade -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 07:59:32.154301 6642 consensus_peers.cc:485] T 8454f24b0c28451b8ee51be39ca3e07f P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 07:59:32.175136 6380 consensus_peers.cc:485] T 782e37a4ebb34a68b461a74115ae8703 P 8315d22c56f7439a9e1fc82f153f018c -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 07:59:32.227820 6249 consensus_peers.cc:485] T f68dbc1f65eb4f1b91a62e2de82a7676 P 8b8b3ad3aa8f4053b2167dd550f2086f -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 07:59:32.258756 6380 consensus_peers.cc:485] T bf806039d8c742d8a80ad994829bdf18 P 8315d22c56f7439a9e1fc82f153f018c -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 07:59:32.286883 6380 consensus_peers.cc:485] T 782e37a4ebb34a68b461a74115ae8703 P 8315d22c56f7439a9e1fc82f153f018c -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 07:59:32.310911 6249 consensus_peers.cc:485] T 2e540f389b2a4aef93a020f549d11b67 P 8b8b3ad3aa8f4053b2167dd550f2086f -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 07:59:32.330986 6642 consensus_peers.cc:462] T a29720541b7b4a79aaa810592ef7c4b1 P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111)
W20240501 07:59:32.370884 6380 consensus_peers.cc:485] T cfcc814f4d2d488ba745ecaf0cc6d61a P 8315d22c56f7439a9e1fc82f153f018c -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 07:59:32.372620 6380 consensus_peers.cc:485] T 9461ad0f65e1489ca102f2558194ce8f P 8315d22c56f7439a9e1fc82f153f018c -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 07:59:32.377579 6511 consensus_peers.cc:462] T 0b24f67cf1a64debb23ee6618c820d33 P 3b164cbbc00f4e8eb3254ddd349a3ff9 -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111)
W20240501 07:59:32.405135 6380 consensus_peers.cc:485] T a756205204d146ce82a2799e30582f57 P 8315d22c56f7439a9e1fc82f153f018c -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 07:59:32.446128 8859 consensus_peers.cc:485] T 0864c290fafa488abefa93d41c0a2efe P 13d05ebdca8c4f489c597ce550dd773d -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 07:59:32.456117 8859 consensus_peers.cc:485] T e0583fa43994452ca7553cbf2309eba1 P 13d05ebdca8c4f489c597ce550dd773d -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 07:59:32.482489 8859 consensus_peers.cc:485] T ad3e528eaff349a4b3b03037652b1505 P 13d05ebdca8c4f489c597ce550dd773d -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 07:59:32.493676 6511 consensus_peers.cc:485] T d3f2250a62c6429e886753aaffcbe883 P 3b164cbbc00f4e8eb3254ddd349a3ff9 -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 07:59:32.538201 8728 consensus_peers.cc:485] T 36aaf192042f4320b2526322424dc640 P 74077b61d6f54342bf9736b4a9f2eade -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 07:59:32.695757 8728 consensus_peers.cc:485] T 6dc929e2a6ae45bd8b1f297583636bb6 P 74077b61d6f54342bf9736b4a9f2eade -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 07:59:32.710428 6642 consensus_peers.cc:485] T 6daebcfc41754ec4aca939ca20588854 P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 07:59:32.716627 6642 consensus_peers.cc:485] T 28eaf22342ff4b5394f958d0b659273c P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 07:59:32.835124 6511 consensus_peers.cc:485] T d3f2250a62c6429e886753aaffcbe883 P 3b164cbbc00f4e8eb3254ddd349a3ff9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 07:59:32.912371 6511 consensus_peers.cc:462] T 0b24f67cf1a64debb23ee6618c820d33 P 3b164cbbc00f4e8eb3254ddd349a3ff9 -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111)
W20240501 07:59:32.918210 6249 consensus_peers.cc:485] T 1c32cd00bc344ae1b161d3f9b43b53d3 P 8b8b3ad3aa8f4053b2167dd550f2086f -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 07:59:32.927676 6642 consensus_peers.cc:462] T a29720541b7b4a79aaa810592ef7c4b1 P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111)
W20240501 07:59:32.942711 8859 consensus_peers.cc:485] T f0c1c05110894e39b41e6b5c5894b8d8 P 13d05ebdca8c4f489c597ce550dd773d -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 07:59:33.089910 8859 consensus_peers.cc:485] T 8a39f168c8f846cd9b8c4e2f2cf62fdd P 13d05ebdca8c4f489c597ce550dd773d -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 07:59:33.191998 8728 consensus_peers.cc:485] T e5bd3a8f7a7640c09774f781009eb19c P 74077b61d6f54342bf9736b4a9f2eade -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 07:59:33.209144 8859 consensus_peers.cc:485] T b4a2db0198774bb2b4f3df769260e32f P 13d05ebdca8c4f489c597ce550dd773d -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 07:59:33.290877 8859 consensus_peers.cc:485] T f788de4df5454d00ac9bb883f0abc9bd P 13d05ebdca8c4f489c597ce550dd773d -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 07:59:33.320478 8859 consensus_peers.cc:485] T c16b7962c18d4e02b850c7c7c678eae4 P 13d05ebdca8c4f489c597ce550dd773d -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 07:59:33.344640 14242 raft_consensus.cc:482] T c86c420310ec42ffa5b24304fba74267 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83f20e84af4c40d09a4804b6b26e697f)
I20240501 07:59:33.344830 14242 raft_consensus.cc:504] T c86c420310ec42ffa5b24304fba74267 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { replace: true } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: true } }
I20240501 07:59:33.345275 14242 leader_election.cc:288] T c86c420310ec42ffa5b24304fba74267 P 8315d22c56f7439a9e1fc82f153f018c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149), a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607)
W20240501 07:59:33.346670 6380 leader_election.cc:334] T c86c420310ec42ffa5b24304fba74267 P 8315d22c56f7439a9e1fc82f153f018c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111)
W20240501 07:59:33.346809 6380 leader_election.cc:334] T c86c420310ec42ffa5b24304fba74267 P 8315d22c56f7439a9e1fc82f153f018c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111)
I20240501 07:59:33.346871 6380 leader_election.cc:302] T c86c420310ec42ffa5b24304fba74267 P 8315d22c56f7439a9e1fc82f153f018c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8315d22c56f7439a9e1fc82f153f018c; no voters: 83f20e84af4c40d09a4804b6b26e697f, a505e7cb133c4392bfda8a4a20874ae9
I20240501 07:59:33.347157 14242 raft_consensus.cc:2726] T c86c420310ec42ffa5b24304fba74267 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 07:59:33.410519 6511 consensus_peers.cc:462] T 0b24f67cf1a64debb23ee6618c820d33 P 3b164cbbc00f4e8eb3254ddd349a3ff9 -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111)
W20240501 07:59:33.419869 6642 consensus_peers.cc:462] T a29720541b7b4a79aaa810592ef7c4b1 P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111)
W20240501 07:59:33.432062 8859 consensus_peers.cc:485] T 12ce6a21d06c4ba889542758be9dc62d P 13d05ebdca8c4f489c597ce550dd773d -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 07:59:33.453446 8728 consensus_peers.cc:485] T 0f9e2549fe294a93b3b25081a15b2028 P 74077b61d6f54342bf9736b4a9f2eade -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 07:59:33.658571 6511 consensus_peers.cc:485] T af718dc43add4edfac703013655699f4 P 3b164cbbc00f4e8eb3254ddd349a3ff9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 07:59:33.674863 6249 consensus_peers.cc:485] T 5ee738571af740c1b494d4f17413d1d8 P 8b8b3ad3aa8f4053b2167dd550f2086f -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 07:59:33.766163 6380 consensus_peers.cc:485] T 72aa4dc7664b48b7a73d34feba7d44d3 P 8315d22c56f7439a9e1fc82f153f018c -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 07:59:33.766815 8859 consensus_peers.cc:485] T ff9dc384dbac47dc9795617c64688943 P 13d05ebdca8c4f489c597ce550dd773d -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 07:59:33.811916 6511 consensus_peers.cc:485] T 6e2ac8c8857d45b88d3b51147d2609aa P 3b164cbbc00f4e8eb3254ddd349a3ff9 -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 07:59:33.908349 8859 consensus_peers.cc:485] T bab84387a40d40c3b8af7f09eb462886 P 13d05ebdca8c4f489c597ce550dd773d -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 07:59:33.912130 6642 consensus_peers.cc:485] T 404bb81b9c404dd288d2f8160a4273da P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 07:59:33.919178 6511 consensus_peers.cc:485] T 114b6dbafa1c4c44ac42f1d2293e3125 P 3b164cbbc00f4e8eb3254ddd349a3ff9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 07:59:33.950831 6642 consensus_peers.cc:462] T a29720541b7b4a79aaa810592ef7c4b1 P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111)
W20240501 07:59:33.956269 6511 consensus_peers.cc:462] T 0b24f67cf1a64debb23ee6618c820d33 P 3b164cbbc00f4e8eb3254ddd349a3ff9 -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111)
W20240501 07:59:33.973165 6249 consensus_peers.cc:485] T b7d57b7fd5234f50afbc859bc997f3e1 P 8b8b3ad3aa8f4053b2167dd550f2086f -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 07:59:34.034827 6380 consensus_peers.cc:485] T 4664b04b23294ad4b0a0e5c87fdca2ad P 8315d22c56f7439a9e1fc82f153f018c -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 07:59:34.050340 8859 consensus_peers.cc:485] T fb6b46072a8d46e18b53b9f8de702859 P 13d05ebdca8c4f489c597ce550dd773d -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 07:59:34.075989 6380 consensus_peers.cc:485] T c3a9036ccfee45498bca9681518d3699 P 8315d22c56f7439a9e1fc82f153f018c -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 07:59:34.097697 8859 consensus_peers.cc:485] T 239ee596e62d47169eeb6670dc62d59c P 13d05ebdca8c4f489c597ce550dd773d -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 07:59:34.125500 6642 consensus_peers.cc:485] T 4bc915ffbbf345cbbe1195f8dd3ab874 P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 07:59:34.128401 14244 raft_consensus.cc:482] T 7dbbfc5d5848486f983cffdd2f36cc91 P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:34.128541 14244 raft_consensus.cc:504] T 7dbbfc5d5848486f983cffdd2f36cc91 P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } }
I20240501 07:59:34.128945 14244 leader_election.cc:288] T 7dbbfc5d5848486f983cffdd2f36cc91 P 74077b61d6f54342bf9736b4a9f2eade [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149), a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607)
W20240501 07:59:34.130153 8728 leader_election.cc:334] T 7dbbfc5d5848486f983cffdd2f36cc91 P 74077b61d6f54342bf9736b4a9f2eade [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111)
W20240501 07:59:34.130280 8728 leader_election.cc:334] T 7dbbfc5d5848486f983cffdd2f36cc91 P 74077b61d6f54342bf9736b4a9f2eade [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111)
I20240501 07:59:34.130340 8728 leader_election.cc:302] T 7dbbfc5d5848486f983cffdd2f36cc91 P 74077b61d6f54342bf9736b4a9f2eade [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74077b61d6f54342bf9736b4a9f2eade; no voters: 83f20e84af4c40d09a4804b6b26e697f, a505e7cb133c4392bfda8a4a20874ae9
I20240501 07:59:34.130481 14244 raft_consensus.cc:2726] T 7dbbfc5d5848486f983cffdd2f36cc91 P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 07:59:34.157829 8859 consensus_peers.cc:485] T 8f36eefcda3f475d9571e92fdeb52eb4 P 13d05ebdca8c4f489c597ce550dd773d -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 07:59:34.184155 6249 consensus_peers.cc:485] T 0fc1ce00a2024e4a9c19abcc7d181efd P 8b8b3ad3aa8f4053b2167dd550f2086f -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 07:59:34.222957 6249 consensus_peers.cc:485] T 6c3c0d6fe41f4ae196ae29beb9b68125 P 8b8b3ad3aa8f4053b2167dd550f2086f -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 07:59:34.227039 14252 raft_consensus.cc:482] T 2976d6353e8947b7b6c37f72efbeeac4 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83f20e84af4c40d09a4804b6b26e697f)
I20240501 07:59:34.227205 14252 raft_consensus.cc:504] T 2976d6353e8947b7b6c37f72efbeeac4 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: false } }
I20240501 07:59:34.227587 14252 leader_election.cc:288] T 2976d6353e8947b7b6c37f72efbeeac4 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607), 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149)
W20240501 07:59:34.228770 6511 leader_election.cc:334] T 2976d6353e8947b7b6c37f72efbeeac4 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111)
W20240501 07:59:34.229200 6511 leader_election.cc:334] T 2976d6353e8947b7b6c37f72efbeeac4 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111)
I20240501 07:59:34.229279 6511 leader_election.cc:302] T 2976d6353e8947b7b6c37f72efbeeac4 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b164cbbc00f4e8eb3254ddd349a3ff9; no voters: 83f20e84af4c40d09a4804b6b26e697f, a505e7cb133c4392bfda8a4a20874ae9
I20240501 07:59:34.229447 14252 raft_consensus.cc:2726] T 2976d6353e8947b7b6c37f72efbeeac4 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 07:59:34.229899 8859 consensus_peers.cc:485] T c197ba256dc74e6d92e34799c274b4e3 P 13d05ebdca8c4f489c597ce550dd773d -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 07:59:34.256295 6642 consensus_peers.cc:485] T 3b9b71d44f754353b2321e8c36dd4dfd P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 07:59:34.300215 6380 consensus_peers.cc:485] T f88fe033bce14bf0b1b63a52a7ba137e P 8315d22c56f7439a9e1fc82f153f018c -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 07:59:34.305536 6642 consensus_peers.cc:485] T 2560b1ec81ac4357b32c7a8ba9d72216 P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 07:59:34.334555 6511 consensus_peers.cc:462] T 0b24f67cf1a64debb23ee6618c820d33 P 3b164cbbc00f4e8eb3254ddd349a3ff9 -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111)
W20240501 07:59:34.387099 8859 consensus_peers.cc:485] T e0583fa43994452ca7553cbf2309eba1 P 13d05ebdca8c4f489c597ce550dd773d -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 07:59:34.397049 6380 consensus_peers.cc:485] T bf806039d8c742d8a80ad994829bdf18 P 8315d22c56f7439a9e1fc82f153f018c -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 07:59:34.402925 8859 consensus_peers.cc:485] T c197ba256dc74e6d92e34799c274b4e3 P 13d05ebdca8c4f489c597ce550dd773d -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 07:59:34.407089 6249 consensus_peers.cc:485] T 2e540f389b2a4aef93a020f549d11b67 P 8b8b3ad3aa8f4053b2167dd550f2086f -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 07:59:34.430433 8728 consensus_peers.cc:485] T fcb3198c1fa5449781dc84756fe02bff P 74077b61d6f54342bf9736b4a9f2eade -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 07:59:34.459110 6642 consensus_peers.cc:485] T a29720541b7b4a79aaa810592ef7c4b1 P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 07:59:34.523450 6642 consensus_peers.cc:462] T a29720541b7b4a79aaa810592ef7c4b1 P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111)
W20240501 07:59:34.525764 8728 consensus_peers.cc:485] T ab1666f1e5204416adb435d9bbe80433 P 74077b61d6f54342bf9736b4a9f2eade -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 07:59:34.624800 6380 consensus_peers.cc:485] T d726533fb72d4827991e12034b42634e P 8315d22c56f7439a9e1fc82f153f018c -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 07:59:34.660715 6380 consensus_peers.cc:485] T 782e37a4ebb34a68b461a74115ae8703 P 8315d22c56f7439a9e1fc82f153f018c -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 07:59:34.669381 8728 consensus_peers.cc:485] T c5a876dfb598428ab350593129505504 P 74077b61d6f54342bf9736b4a9f2eade -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 07:59:34.676363 8859 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111) [suppressed 170 similar messages]
W20240501 07:59:34.722936 6511 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111) [suppressed 85 similar messages]
W20240501 07:59:34.723603 6511 consensus_peers.cc:485] T 3745acee2b5543b697a3c8f45d2618bb P 3b164cbbc00f4e8eb3254ddd349a3ff9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 07:59:34.754696 6642 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111) [suppressed 85 similar messages]
W20240501 07:59:34.757906 8728 consensus_peers.cc:485] T 36aaf192042f4320b2526322424dc640 P 74077b61d6f54342bf9736b4a9f2eade -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 07:59:34.806993 6380 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111) [suppressed 124 similar messages]
W20240501 07:59:34.815392 6380 consensus_peers.cc:485] T 9461ad0f65e1489ca102f2558194ce8f P 8315d22c56f7439a9e1fc82f153f018c -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 07:59:34.840821 6642 consensus_peers.cc:485] T 8454f24b0c28451b8ee51be39ca3e07f P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 07:59:34.841112 6380 consensus_peers.cc:485] T a756205204d146ce82a2799e30582f57 P 8315d22c56f7439a9e1fc82f153f018c -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 07:59:34.845319 6511 consensus_peers.cc:485] T d3f2250a62c6429e886753aaffcbe883 P 3b164cbbc00f4e8eb3254ddd349a3ff9 -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 07:59:34.846159 14252 raft_consensus.cc:482] T e81b23661a1d475887c63f79d8f90ab6 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:34.846252 14252 raft_consensus.cc:504] T e81b23661a1d475887c63f79d8f90ab6 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } }
I20240501 07:59:34.846616 14252 leader_election.cc:288] T e81b23661a1d475887c63f79d8f90ab6 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149), a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607)
W20240501 07:59:34.847576 6511 leader_election.cc:334] T e81b23661a1d475887c63f79d8f90ab6 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111)
W20240501 07:59:34.847673 6511 leader_election.cc:334] T e81b23661a1d475887c63f79d8f90ab6 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111)
I20240501 07:59:34.847707 6511 leader_election.cc:302] T e81b23661a1d475887c63f79d8f90ab6 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b164cbbc00f4e8eb3254ddd349a3ff9; no voters: 83f20e84af4c40d09a4804b6b26e697f, a505e7cb133c4392bfda8a4a20874ae9
I20240501 07:59:34.847823 14252 raft_consensus.cc:2726] T e81b23661a1d475887c63f79d8f90ab6 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240501 07:59:34.853467 6511 consensus_peers.cc:462] T 0b24f67cf1a64debb23ee6618c820d33 P 3b164cbbc00f4e8eb3254ddd349a3ff9 -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111)
W20240501 07:59:34.863535 8728 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111) [suppressed 87 similar messages]
W20240501 07:59:34.864146 8728 consensus_peers.cc:485] T 6dc929e2a6ae45bd8b1f297583636bb6 P 74077b61d6f54342bf9736b4a9f2eade -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 07:59:34.866497 6380 consensus_peers.cc:485] T bf806039d8c742d8a80ad994829bdf18 P 8315d22c56f7439a9e1fc82f153f018c -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 07:59:34.868566 6249 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240501 07:59:34.906666 6380 consensus_peers.cc:485] T cfcc814f4d2d488ba745ecaf0cc6d61a P 8315d22c56f7439a9e1fc82f153f018c -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 07:59:34.938477 6249 consensus_peers.cc:485] T f68dbc1f65eb4f1b91a62e2de82a7676 P 8b8b3ad3aa8f4053b2167dd550f2086f -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 07:59:34.943493 6380 consensus_peers.cc:485] T 782e37a4ebb34a68b461a74115ae8703 P 8315d22c56f7439a9e1fc82f153f018c -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 07:59:34.949272 6511 consensus_peers.cc:485] T d6c28cd91f624673b8e428f471a1dcce P 3b164cbbc00f4e8eb3254ddd349a3ff9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 07:59:34.953285 8728 consensus_peers.cc:485] T f3375300a7914fa599fe0d3a8f1f9c18 P 74077b61d6f54342bf9736b4a9f2eade -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 07:59:34.972929 8859 consensus_peers.cc:485] T e0583fa43994452ca7553cbf2309eba1 P 13d05ebdca8c4f489c597ce550dd773d -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 07:59:35.085026 6642 consensus_peers.cc:462] T a29720541b7b4a79aaa810592ef7c4b1 P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111)
W20240501 07:59:35.092124 8859 consensus_peers.cc:485] T 0864c290fafa488abefa93d41c0a2efe P 13d05ebdca8c4f489c597ce550dd773d -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 07:59:35.239617 6511 consensus_peers.cc:485] T d3f2250a62c6429e886753aaffcbe883 P 3b164cbbc00f4e8eb3254ddd349a3ff9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 07:59:35.254582 6511 consensus_peers.cc:462] T 0b24f67cf1a64debb23ee6618c820d33 P 3b164cbbc00f4e8eb3254ddd349a3ff9 -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111)
W20240501 07:59:35.321182 8859 consensus_peers.cc:485] T ad3e528eaff349a4b3b03037652b1505 P 13d05ebdca8c4f489c597ce550dd773d -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 07:59:35.436256 6642 consensus_peers.cc:485] T 28eaf22342ff4b5394f958d0b659273c P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 07:59:35.463274 6249 consensus_peers.cc:485] T 1c32cd00bc344ae1b161d3f9b43b53d3 P 8b8b3ad3aa8f4053b2167dd550f2086f -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 07:59:35.483711 8859 consensus_peers.cc:485] T f0c1c05110894e39b41e6b5c5894b8d8 P 13d05ebdca8c4f489c597ce550dd773d -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 07:59:35.533537 6642 consensus_peers.cc:485] T 6daebcfc41754ec4aca939ca20588854 P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 07:59:35.622203 6642 consensus_peers.cc:462] T a29720541b7b4a79aaa810592ef7c4b1 P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111)
W20240501 07:59:35.640997 8859 consensus_peers.cc:485] T 8a39f168c8f846cd9b8c4e2f2cf62fdd P 13d05ebdca8c4f489c597ce550dd773d -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 07:59:35.658324 6511 consensus_peers.cc:462] T 0b24f67cf1a64debb23ee6618c820d33 P 3b164cbbc00f4e8eb3254ddd349a3ff9 -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111)
W20240501 07:59:35.735591 8859 consensus_peers.cc:485] T c16b7962c18d4e02b850c7c7c678eae4 P 13d05ebdca8c4f489c597ce550dd773d -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 07:59:35.903252 8859 consensus_peers.cc:485] T b4a2db0198774bb2b4f3df769260e32f P 13d05ebdca8c4f489c597ce550dd773d -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 07:59:35.919842 8728 consensus_peers.cc:485] T e5bd3a8f7a7640c09774f781009eb19c P 74077b61d6f54342bf9736b4a9f2eade -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 07:59:35.940131 8859 consensus_peers.cc:485] T f788de4df5454d00ac9bb883f0abc9bd P 13d05ebdca8c4f489c597ce550dd773d -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 07:59:35.975800 8728 consensus_peers.cc:485] T 0f9e2549fe294a93b3b25081a15b2028 P 74077b61d6f54342bf9736b4a9f2eade -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 07:59:35.988685 6511 consensus_peers.cc:485] T af718dc43add4edfac703013655699f4 P 3b164cbbc00f4e8eb3254ddd349a3ff9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 07:59:36.020756 6642 consensus_peers.cc:462] T a29720541b7b4a79aaa810592ef7c4b1 P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111)
W20240501 07:59:36.022434 8859 consensus_peers.cc:485] T 12ce6a21d06c4ba889542758be9dc62d P 13d05ebdca8c4f489c597ce550dd773d -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 07:59:36.161033 14266 raft_consensus.cc:482] T 13daf8496cd64d7d959ce5ad666b9be8 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a505e7cb133c4392bfda8a4a20874ae9)
I20240501 07:59:36.161167 14266 raft_consensus.cc:504] T 13daf8496cd64d7d959ce5ad666b9be8 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } }
I20240501 07:59:36.161942 14266 leader_election.cc:288] T 13daf8496cd64d7d959ce5ad666b9be8 P 8315d22c56f7439a9e1fc82f153f018c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149), a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607)
W20240501 07:59:36.162883 6380 leader_election.cc:334] T 13daf8496cd64d7d959ce5ad666b9be8 P 8315d22c56f7439a9e1fc82f153f018c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111)
W20240501 07:59:36.163004 6380 leader_election.cc:334] T 13daf8496cd64d7d959ce5ad666b9be8 P 8315d22c56f7439a9e1fc82f153f018c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111)
I20240501 07:59:36.163060 6380 leader_election.cc:302] T 13daf8496cd64d7d959ce5ad666b9be8 P 8315d22c56f7439a9e1fc82f153f018c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8315d22c56f7439a9e1fc82f153f018c; no voters: 83f20e84af4c40d09a4804b6b26e697f, a505e7cb133c4392bfda8a4a20874ae9
I20240501 07:59:36.163204 14266 raft_consensus.cc:2726] T 13daf8496cd64d7d959ce5ad666b9be8 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 07:59:36.172381 6511 consensus_peers.cc:462] T 0b24f67cf1a64debb23ee6618c820d33 P 3b164cbbc00f4e8eb3254ddd349a3ff9 -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111)
W20240501 07:59:36.186069 6249 consensus_peers.cc:485] T 5ee738571af740c1b494d4f17413d1d8 P 8b8b3ad3aa8f4053b2167dd550f2086f -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 07:59:36.261061 6511 consensus_peers.cc:485] T 6e2ac8c8857d45b88d3b51147d2609aa P 3b164cbbc00f4e8eb3254ddd349a3ff9 -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 07:59:36.292166 6380 consensus_peers.cc:485] T 72aa4dc7664b48b7a73d34feba7d44d3 P 8315d22c56f7439a9e1fc82f153f018c -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 07:59:36.294500 6158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111) [suppressed 12 similar messages]
W20240501 07:59:36.295188 6158 catalog_manager.cc:4573] TS 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): CreateTablet RPC failed for tablet 7dbbfc5d5848486f983cffdd2f36cc91: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111)
W20240501 07:59:36.360859 6380 consensus_peers.cc:485] T 4664b04b23294ad4b0a0e5c87fdca2ad P 8315d22c56f7439a9e1fc82f153f018c -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 07:59:36.371013 6511 consensus_peers.cc:485] T 114b6dbafa1c4c44ac42f1d2293e3125 P 3b164cbbc00f4e8eb3254ddd349a3ff9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 07:59:36.379925 6249 consensus_peers.cc:485] T b7d57b7fd5234f50afbc859bc997f3e1 P 8b8b3ad3aa8f4053b2167dd550f2086f -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 07:59:36.388372 8859 consensus_peers.cc:485] T bab84387a40d40c3b8af7f09eb462886 P 13d05ebdca8c4f489c597ce550dd773d -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 07:59:36.398653 6642 consensus_peers.cc:462] T a29720541b7b4a79aaa810592ef7c4b1 P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111)
W20240501 07:59:36.469866 6642 consensus_peers.cc:485] T 3b9b71d44f754353b2321e8c36dd4dfd P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 07:59:36.522863 8859 consensus_peers.cc:485] T ff9dc384dbac47dc9795617c64688943 P 13d05ebdca8c4f489c597ce550dd773d -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 07:59:36.556403 6642 consensus_peers.cc:485] T 404bb81b9c404dd288d2f8160a4273da P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 07:59:36.609526 8859 consensus_peers.cc:485] T c197ba256dc74e6d92e34799c274b4e3 P 13d05ebdca8c4f489c597ce550dd773d -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 07:59:36.631876 8859 consensus_peers.cc:485] T fb6b46072a8d46e18b53b9f8de702859 P 13d05ebdca8c4f489c597ce550dd773d -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 07:59:36.650472 6249 consensus_peers.cc:485] T 0fc1ce00a2024e4a9c19abcc7d181efd P 8b8b3ad3aa8f4053b2167dd550f2086f -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 07:59:36.660183 6380 consensus_peers.cc:485] T c3a9036ccfee45498bca9681518d3699 P 8315d22c56f7439a9e1fc82f153f018c -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 07:59:36.670711 6249 consensus_peers.cc:485] T 6c3c0d6fe41f4ae196ae29beb9b68125 P 8b8b3ad3aa8f4053b2167dd550f2086f -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 07:59:36.672462 8859 consensus_peers.cc:485] T 239ee596e62d47169eeb6670dc62d59c P 13d05ebdca8c4f489c597ce550dd773d -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 07:59:36.686129 6380 consensus_peers.cc:485] T f88fe033bce14bf0b1b63a52a7ba137e P 8315d22c56f7439a9e1fc82f153f018c -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 07:59:36.715397 6511 consensus_peers.cc:462] T 0b24f67cf1a64debb23ee6618c820d33 P 3b164cbbc00f4e8eb3254ddd349a3ff9 -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111)
W20240501 07:59:36.757952 8859 consensus_peers.cc:485] T 8f36eefcda3f475d9571e92fdeb52eb4 P 13d05ebdca8c4f489c597ce550dd773d -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 07:59:36.785710 8859 consensus_peers.cc:485] T c197ba256dc74e6d92e34799c274b4e3 P 13d05ebdca8c4f489c597ce550dd773d -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 07:59:36.867619 8728 consensus_peers.cc:485] T ab1666f1e5204416adb435d9bbe80433 P 74077b61d6f54342bf9736b4a9f2eade -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 07:59:36.903334 6249 consensus_peers.cc:485] T 2e540f389b2a4aef93a020f549d11b67 P 8b8b3ad3aa8f4053b2167dd550f2086f -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 07:59:36.907325 6642 consensus_peers.cc:485] T 2560b1ec81ac4357b32c7a8ba9d72216 P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 07:59:36.957439 6642 consensus_peers.cc:485] T 4bc915ffbbf345cbbe1195f8dd3ab874 P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 07:59:36.969987 6642 consensus_peers.cc:462] T a29720541b7b4a79aaa810592ef7c4b1 P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111)
W20240501 07:59:37.002851 8859 consensus_peers.cc:485] T e0583fa43994452ca7553cbf2309eba1 P 13d05ebdca8c4f489c597ce550dd773d -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 07:59:37.006891 14269 raft_consensus.cc:482] T c86c420310ec42ffa5b24304fba74267 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83f20e84af4c40d09a4804b6b26e697f)
I20240501 07:59:37.007037 14269 raft_consensus.cc:504] T c86c420310ec42ffa5b24304fba74267 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { replace: true } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: true } }
I20240501 07:59:37.007428 14269 leader_election.cc:288] T c86c420310ec42ffa5b24304fba74267 P 8315d22c56f7439a9e1fc82f153f018c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149), a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607)
W20240501 07:59:37.008427 6380 leader_election.cc:334] T c86c420310ec42ffa5b24304fba74267 P 8315d22c56f7439a9e1fc82f153f018c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111)
W20240501 07:59:37.008546 6380 leader_election.cc:334] T c86c420310ec42ffa5b24304fba74267 P 8315d22c56f7439a9e1fc82f153f018c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111)
I20240501 07:59:37.008605 6380 leader_election.cc:302] T c86c420310ec42ffa5b24304fba74267 P 8315d22c56f7439a9e1fc82f153f018c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8315d22c56f7439a9e1fc82f153f018c; no voters: 83f20e84af4c40d09a4804b6b26e697f, a505e7cb133c4392bfda8a4a20874ae9
I20240501 07:59:37.008823 14269 raft_consensus.cc:2726] T c86c420310ec42ffa5b24304fba74267 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 07:59:37.089625 14272 raft_consensus.cc:482] T e81b23661a1d475887c63f79d8f90ab6 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:37.089769 14272 raft_consensus.cc:504] T e81b23661a1d475887c63f79d8f90ab6 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } }
I20240501 07:59:37.090224 14272 leader_election.cc:288] T e81b23661a1d475887c63f79d8f90ab6 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149), a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607)
W20240501 07:59:37.091295 6511 leader_election.cc:334] T e81b23661a1d475887c63f79d8f90ab6 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111)
W20240501 07:59:37.091403 6511 leader_election.cc:334] T e81b23661a1d475887c63f79d8f90ab6 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111)
I20240501 07:59:37.091457 6511 leader_election.cc:302] T e81b23661a1d475887c63f79d8f90ab6 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b164cbbc00f4e8eb3254ddd349a3ff9; no voters: 83f20e84af4c40d09a4804b6b26e697f, a505e7cb133c4392bfda8a4a20874ae9
I20240501 07:59:37.091610 14272 raft_consensus.cc:2726] T e81b23661a1d475887c63f79d8f90ab6 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240501 07:59:37.124941 6380 consensus_peers.cc:485] T bf806039d8c742d8a80ad994829bdf18 P 8315d22c56f7439a9e1fc82f153f018c -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 07:59:37.126919 6511 consensus_peers.cc:462] T 0b24f67cf1a64debb23ee6618c820d33 P 3b164cbbc00f4e8eb3254ddd349a3ff9 -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111)
W20240501 07:59:37.130831 6642 consensus_peers.cc:485] T a29720541b7b4a79aaa810592ef7c4b1 P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 07:59:37.133306 6380 consensus_peers.cc:485] T 782e37a4ebb34a68b461a74115ae8703 P 8315d22c56f7439a9e1fc82f153f018c -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 07:59:37.150683 6380 consensus_peers.cc:485] T 9461ad0f65e1489ca102f2558194ce8f P 8315d22c56f7439a9e1fc82f153f018c -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 07:59:37.151873 8728 consensus_peers.cc:485] T fcb3198c1fa5449781dc84756fe02bff P 74077b61d6f54342bf9736b4a9f2eade -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 07:59:37.177137 6249 consensus_peers.cc:485] T f68dbc1f65eb4f1b91a62e2de82a7676 P 8b8b3ad3aa8f4053b2167dd550f2086f -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 07:59:37.217340 6642 consensus_peers.cc:485] T 8454f24b0c28451b8ee51be39ca3e07f P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 07:59:37.223608 8728 consensus_peers.cc:485] T c5a876dfb598428ab350593129505504 P 74077b61d6f54342bf9736b4a9f2eade -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 07:59:37.247531 8728 consensus_peers.cc:485] T 36aaf192042f4320b2526322424dc640 P 74077b61d6f54342bf9736b4a9f2eade -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 07:59:37.293826 6511 consensus_peers.cc:485] T 3745acee2b5543b697a3c8f45d2618bb P 3b164cbbc00f4e8eb3254ddd349a3ff9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 07:59:37.365578 6380 consensus_peers.cc:485] T a756205204d146ce82a2799e30582f57 P 8315d22c56f7439a9e1fc82f153f018c -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 07:59:37.396502 6380 consensus_peers.cc:485] T d726533fb72d4827991e12034b42634e P 8315d22c56f7439a9e1fc82f153f018c -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 07:59:37.417732 6380 consensus_peers.cc:485] T cfcc814f4d2d488ba745ecaf0cc6d61a P 8315d22c56f7439a9e1fc82f153f018c -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 07:59:37.449173 6511 consensus_peers.cc:485] T d3f2250a62c6429e886753aaffcbe883 P 3b164cbbc00f4e8eb3254ddd349a3ff9 -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 07:59:37.478183 8728 consensus_peers.cc:485] T f3375300a7914fa599fe0d3a8f1f9c18 P 74077b61d6f54342bf9736b4a9f2eade -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 07:59:37.478595 6642 consensus_peers.cc:462] T a29720541b7b4a79aaa810592ef7c4b1 P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111)
W20240501 07:59:37.490453 6511 consensus_peers.cc:485] T d6c28cd91f624673b8e428f471a1dcce P 3b164cbbc00f4e8eb3254ddd349a3ff9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 07:59:37.526713 8728 consensus_peers.cc:485] T 6dc929e2a6ae45bd8b1f297583636bb6 P 74077b61d6f54342bf9736b4a9f2eade -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 07:59:37.543607 6511 consensus_peers.cc:462] T 0b24f67cf1a64debb23ee6618c820d33 P 3b164cbbc00f4e8eb3254ddd349a3ff9 -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111)
W20240501 07:59:37.560808 8859 consensus_peers.cc:485] T e0583fa43994452ca7553cbf2309eba1 P 13d05ebdca8c4f489c597ce550dd773d -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 07:59:37.582371 6380 consensus_peers.cc:485] T bf806039d8c742d8a80ad994829bdf18 P 8315d22c56f7439a9e1fc82f153f018c -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 07:59:37.602108 6380 consensus_peers.cc:485] T 782e37a4ebb34a68b461a74115ae8703 P 8315d22c56f7439a9e1fc82f153f018c -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 07:59:37.631922 8859 consensus_peers.cc:485] T 0864c290fafa488abefa93d41c0a2efe P 13d05ebdca8c4f489c597ce550dd773d -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 07:59:37.668216 6158 catalog_manager.cc:4573] TS a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): CreateTablet RPC failed for tablet e81b23661a1d475887c63f79d8f90ab6: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111)
W20240501 07:59:37.739514 8859 consensus_peers.cc:485] T ad3e528eaff349a4b3b03037652b1505 P 13d05ebdca8c4f489c597ce550dd773d -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 07:59:37.743860 6511 consensus_peers.cc:485] T d3f2250a62c6429e886753aaffcbe883 P 3b164cbbc00f4e8eb3254ddd349a3ff9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 07:59:37.923539 6642 consensus_peers.cc:462] T a29720541b7b4a79aaa810592ef7c4b1 P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111)
W20240501 07:59:37.927775 6249 consensus_peers.cc:485] T 1c32cd00bc344ae1b161d3f9b43b53d3 P 8b8b3ad3aa8f4053b2167dd550f2086f -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 07:59:38.086283 6642 consensus_peers.cc:485] T 6daebcfc41754ec4aca939ca20588854 P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 07:59:38.121347 6511 consensus_peers.cc:462] T 0b24f67cf1a64debb23ee6618c820d33 P 3b164cbbc00f4e8eb3254ddd349a3ff9 -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111)
W20240501 07:59:38.121677 6642 consensus_peers.cc:485] T 28eaf22342ff4b5394f958d0b659273c P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 07:59:38.193384 8859 consensus_peers.cc:485] T 8a39f168c8f846cd9b8c4e2f2cf62fdd P 13d05ebdca8c4f489c597ce550dd773d -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 07:59:38.206537 8859 consensus_peers.cc:485] T f0c1c05110894e39b41e6b5c5894b8d8 P 13d05ebdca8c4f489c597ce550dd773d -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 07:59:38.211284 8859 consensus_peers.cc:485] T b4a2db0198774bb2b4f3df769260e32f P 13d05ebdca8c4f489c597ce550dd773d -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 07:59:38.364496 8859 consensus_peers.cc:485] T c16b7962c18d4e02b850c7c7c678eae4 P 13d05ebdca8c4f489c597ce550dd773d -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 07:59:38.367307 8728 consensus_peers.cc:485] T e5bd3a8f7a7640c09774f781009eb19c P 74077b61d6f54342bf9736b4a9f2eade -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 07:59:38.417618 8859 consensus_peers.cc:485] T 12ce6a21d06c4ba889542758be9dc62d P 13d05ebdca8c4f489c597ce550dd773d -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 07:59:38.429399 6642 consensus_peers.cc:462] T a29720541b7b4a79aaa810592ef7c4b1 P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111)
W20240501 07:59:38.501338 8728 consensus_peers.cc:485] T 0f9e2549fe294a93b3b25081a15b2028 P 74077b61d6f54342bf9736b4a9f2eade -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 07:59:38.530095 6511 consensus_peers.cc:485] T af718dc43add4edfac703013655699f4 P 3b164cbbc00f4e8eb3254ddd349a3ff9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 07:59:38.552527 6511 consensus_peers.cc:462] T 0b24f67cf1a64debb23ee6618c820d33 P 3b164cbbc00f4e8eb3254ddd349a3ff9 -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111)
W20240501 07:59:38.584456 8859 consensus_peers.cc:485] T f788de4df5454d00ac9bb883f0abc9bd P 13d05ebdca8c4f489c597ce550dd773d -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 07:59:38.718770 6380 consensus_peers.cc:485] T 72aa4dc7664b48b7a73d34feba7d44d3 P 8315d22c56f7439a9e1fc82f153f018c -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 07:59:38.774781 6249 consensus_peers.cc:485] T b7d57b7fd5234f50afbc859bc997f3e1 P 8b8b3ad3aa8f4053b2167dd550f2086f -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 07:59:38.804165 14272 raft_consensus.cc:482] T 2976d6353e8947b7b6c37f72efbeeac4 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83f20e84af4c40d09a4804b6b26e697f)
I20240501 07:59:38.804317 14272 raft_consensus.cc:504] T 2976d6353e8947b7b6c37f72efbeeac4 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: false } }
I20240501 07:59:38.804688 14272 leader_election.cc:288] T 2976d6353e8947b7b6c37f72efbeeac4 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607), 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149)
W20240501 07:59:38.805684 6511 leader_election.cc:334] T 2976d6353e8947b7b6c37f72efbeeac4 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111)
W20240501 07:59:38.805795 6511 leader_election.cc:334] T 2976d6353e8947b7b6c37f72efbeeac4 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111)
I20240501 07:59:38.805847 6511 leader_election.cc:302] T 2976d6353e8947b7b6c37f72efbeeac4 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b164cbbc00f4e8eb3254ddd349a3ff9; no voters: 83f20e84af4c40d09a4804b6b26e697f, a505e7cb133c4392bfda8a4a20874ae9
I20240501 07:59:38.805984 14272 raft_consensus.cc:2726] T 2976d6353e8947b7b6c37f72efbeeac4 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 07:59:38.830423 6380 consensus_peers.cc:485] T c3a9036ccfee45498bca9681518d3699 P 8315d22c56f7439a9e1fc82f153f018c -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 07:59:38.896323 6642 consensus_peers.cc:462] T a29720541b7b4a79aaa810592ef7c4b1 P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111)
W20240501 07:59:38.925730 6511 consensus_peers.cc:485] T 6e2ac8c8857d45b88d3b51147d2609aa P 3b164cbbc00f4e8eb3254ddd349a3ff9 -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 07:59:38.966161 6511 consensus_peers.cc:462] T 0b24f67cf1a64debb23ee6618c820d33 P 3b164cbbc00f4e8eb3254ddd349a3ff9 -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111)
W20240501 07:59:38.997619 8859 consensus_peers.cc:485] T c197ba256dc74e6d92e34799c274b4e3 P 13d05ebdca8c4f489c597ce550dd773d -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 07:59:39.003190 6249 consensus_peers.cc:485] T 5ee738571af740c1b494d4f17413d1d8 P 8b8b3ad3aa8f4053b2167dd550f2086f -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 07:59:39.022836 8859 consensus_peers.cc:485] T c197ba256dc74e6d92e34799c274b4e3 P 13d05ebdca8c4f489c597ce550dd773d -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 07:59:39.028582 6642 consensus_peers.cc:485] T 3b9b71d44f754353b2321e8c36dd4dfd P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 07:59:39.080246 8859 consensus_peers.cc:485] T ff9dc384dbac47dc9795617c64688943 P 13d05ebdca8c4f489c597ce550dd773d -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 07:59:39.114970 8859 consensus_peers.cc:485] T bab84387a40d40c3b8af7f09eb462886 P 13d05ebdca8c4f489c597ce550dd773d -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 07:59:39.135891 6642 consensus_peers.cc:485] T 404bb81b9c404dd288d2f8160a4273da P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 07:59:39.149698 6380 consensus_peers.cc:485] T 4664b04b23294ad4b0a0e5c87fdca2ad P 8315d22c56f7439a9e1fc82f153f018c -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 07:59:39.165136 6249 consensus_peers.cc:485] T 6c3c0d6fe41f4ae196ae29beb9b68125 P 8b8b3ad3aa8f4053b2167dd550f2086f -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 07:59:39.166249 8859 consensus_peers.cc:485] T 8f36eefcda3f475d9571e92fdeb52eb4 P 13d05ebdca8c4f489c597ce550dd773d -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 07:59:39.186810 8859 consensus_peers.cc:485] T 239ee596e62d47169eeb6670dc62d59c P 13d05ebdca8c4f489c597ce550dd773d -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 07:59:39.204489 6511 consensus_peers.cc:485] T 114b6dbafa1c4c44ac42f1d2293e3125 P 3b164cbbc00f4e8eb3254ddd349a3ff9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 07:59:39.238313 8859 consensus_peers.cc:485] T fb6b46072a8d46e18b53b9f8de702859 P 13d05ebdca8c4f489c597ce550dd773d -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 07:59:39.322764 6642 consensus_peers.cc:485] T a29720541b7b4a79aaa810592ef7c4b1 P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 07:59:39.420652 6249 consensus_peers.cc:485] T 0fc1ce00a2024e4a9c19abcc7d181efd P 8b8b3ad3aa8f4053b2167dd550f2086f -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 07:59:39.450886 6249 consensus_peers.cc:485] T 2e540f389b2a4aef93a020f549d11b67 P 8b8b3ad3aa8f4053b2167dd550f2086f -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 07:59:39.459688 6642 consensus_peers.cc:462] T a29720541b7b4a79aaa810592ef7c4b1 P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111)
W20240501 07:59:39.460136 6642 consensus_peers.cc:485] T 8454f24b0c28451b8ee51be39ca3e07f P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 07:59:39.466377 6380 consensus_peers.cc:485] T f88fe033bce14bf0b1b63a52a7ba137e P 8315d22c56f7439a9e1fc82f153f018c -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 07:59:39.472390 6249 consensus_peers.cc:485] T f68dbc1f65eb4f1b91a62e2de82a7676 P 8b8b3ad3aa8f4053b2167dd550f2086f -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 07:59:39.481621 6642 consensus_peers.cc:485] T 4bc915ffbbf345cbbe1195f8dd3ab874 P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 07:59:39.492611 8728 consensus_peers.cc:485] T ab1666f1e5204416adb435d9bbe80433 P 74077b61d6f54342bf9736b4a9f2eade -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 07:59:39.536211 6380 consensus_peers.cc:485] T 9461ad0f65e1489ca102f2558194ce8f P 8315d22c56f7439a9e1fc82f153f018c -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 07:59:39.568507 6642 consensus_peers.cc:485] T 2560b1ec81ac4357b32c7a8ba9d72216 P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 07:59:39.571453 8728 consensus_peers.cc:485] T fcb3198c1fa5449781dc84756fe02bff P 74077b61d6f54342bf9736b4a9f2eade -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 07:59:39.590380 6511 consensus_peers.cc:462] T 0b24f67cf1a64debb23ee6618c820d33 P 3b164cbbc00f4e8eb3254ddd349a3ff9 -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111)
W20240501 07:59:39.590595 8859 consensus_peers.cc:485] T e0583fa43994452ca7553cbf2309eba1 P 13d05ebdca8c4f489c597ce550dd773d -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 07:59:39.646797 8728 consensus_peers.cc:485] T c5a876dfb598428ab350593129505504 P 74077b61d6f54342bf9736b4a9f2eade -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 07:59:39.664165 6380 consensus_peers.cc:485] T 782e37a4ebb34a68b461a74115ae8703 P 8315d22c56f7439a9e1fc82f153f018c -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 07:59:39.667115 6380 consensus_peers.cc:485] T bf806039d8c742d8a80ad994829bdf18 P 8315d22c56f7439a9e1fc82f153f018c -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 07:59:39.676549 8859 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111) [suppressed 171 similar messages]
W20240501 07:59:39.724563 6380 consensus_peers.cc:485] T d726533fb72d4827991e12034b42634e P 8315d22c56f7439a9e1fc82f153f018c -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 07:59:39.783866 6511 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111) [suppressed 85 similar messages]
W20240501 07:59:39.784868 6511 consensus_peers.cc:485] T 3745acee2b5543b697a3c8f45d2618bb P 3b164cbbc00f4e8eb3254ddd349a3ff9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 07:59:39.818965 6642 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111) [suppressed 88 similar messages]
W20240501 07:59:39.825765 6380 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111) [suppressed 125 similar messages]
W20240501 07:59:39.826972 6380 consensus_peers.cc:485] T a756205204d146ce82a2799e30582f57 P 8315d22c56f7439a9e1fc82f153f018c -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 07:59:39.864414 8728 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111) [suppressed 78 similar messages]
W20240501 07:59:39.867488 8728 consensus_peers.cc:485] T 36aaf192042f4320b2526322424dc640 P 74077b61d6f54342bf9736b4a9f2eade -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 07:59:39.907496 6511 consensus_peers.cc:485] T d3f2250a62c6429e886753aaffcbe883 P 3b164cbbc00f4e8eb3254ddd349a3ff9 -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 07:59:39.917893 6642 consensus_peers.cc:462] T a29720541b7b4a79aaa810592ef7c4b1 P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111)
W20240501 07:59:39.918314 8859 consensus_peers.cc:485] T e0583fa43994452ca7553cbf2309eba1 P 13d05ebdca8c4f489c597ce550dd773d -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 07:59:39.936496 6249 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240501 07:59:40.000790 6380 consensus_peers.cc:485] T cfcc814f4d2d488ba745ecaf0cc6d61a P 8315d22c56f7439a9e1fc82f153f018c -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 07:59:40.024675 6380 consensus_peers.cc:485] T bf806039d8c742d8a80ad994829bdf18 P 8315d22c56f7439a9e1fc82f153f018c -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 07:59:40.052565 8728 consensus_peers.cc:485] T 6dc929e2a6ae45bd8b1f297583636bb6 P 74077b61d6f54342bf9736b4a9f2eade -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 07:59:40.097676 6511 consensus_peers.cc:485] T d6c28cd91f624673b8e428f471a1dcce P 3b164cbbc00f4e8eb3254ddd349a3ff9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 07:59:40.128892 6511 consensus_peers.cc:462] T 0b24f67cf1a64debb23ee6618c820d33 P 3b164cbbc00f4e8eb3254ddd349a3ff9 -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111)
W20240501 07:59:40.152524 8728 consensus_peers.cc:485] T f3375300a7914fa599fe0d3a8f1f9c18 P 74077b61d6f54342bf9736b4a9f2eade -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 07:59:40.182533 6380 consensus_peers.cc:485] T 782e37a4ebb34a68b461a74115ae8703 P 8315d22c56f7439a9e1fc82f153f018c -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 07:59:40.238020 8859 consensus_peers.cc:485] T 0864c290fafa488abefa93d41c0a2efe P 13d05ebdca8c4f489c597ce550dd773d -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 07:59:40.251578 6511 consensus_peers.cc:485] T d3f2250a62c6429e886753aaffcbe883 P 3b164cbbc00f4e8eb3254ddd349a3ff9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 07:59:40.281504 6249 consensus_peers.cc:485] T 1c32cd00bc344ae1b161d3f9b43b53d3 P 8b8b3ad3aa8f4053b2167dd550f2086f -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 07:59:40.319960 8859 consensus_peers.cc:485] T ad3e528eaff349a4b3b03037652b1505 P 13d05ebdca8c4f489c597ce550dd773d -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 07:59:40.324723 6642 consensus_peers.cc:462] T a29720541b7b4a79aaa810592ef7c4b1 P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111)
I20240501 07:59:40.619511 14264 raft_consensus.cc:482] T a93ce5841c7f40a9ab23f88efe7b85bb P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83f20e84af4c40d09a4804b6b26e697f)
I20240501 07:59:40.619637 14264 raft_consensus.cc:504] T a93ce5841c7f40a9ab23f88efe7b85bb P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { replace: true } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { promote: true } }
I20240501 07:59:40.620013 14264 leader_election.cc:288] T a93ce5841c7f40a9ab23f88efe7b85bb P 74077b61d6f54342bf9736b4a9f2eade [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149), a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607)
W20240501 07:59:40.621138 8728 leader_election.cc:334] T a93ce5841c7f40a9ab23f88efe7b85bb P 74077b61d6f54342bf9736b4a9f2eade [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111)
W20240501 07:59:40.621270 8728 leader_election.cc:334] T a93ce5841c7f40a9ab23f88efe7b85bb P 74077b61d6f54342bf9736b4a9f2eade [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111)
I20240501 07:59:40.621330 8728 leader_election.cc:302] T a93ce5841c7f40a9ab23f88efe7b85bb P 74077b61d6f54342bf9736b4a9f2eade [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74077b61d6f54342bf9736b4a9f2eade; no voters: 83f20e84af4c40d09a4804b6b26e697f, a505e7cb133c4392bfda8a4a20874ae9
I20240501 07:59:40.621460 14264 raft_consensus.cc:2726] T a93ce5841c7f40a9ab23f88efe7b85bb P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 07:59:40.621670 8859 consensus_peers.cc:485] T 12ce6a21d06c4ba889542758be9dc62d P 13d05ebdca8c4f489c597ce550dd773d -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 07:59:40.669219 8859 consensus_peers.cc:485] T 8a39f168c8f846cd9b8c4e2f2cf62fdd P 13d05ebdca8c4f489c597ce550dd773d -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 07:59:40.671154 6642 consensus_peers.cc:485] T 28eaf22342ff4b5394f958d0b659273c P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 07:59:40.675868 8859 consensus_peers.cc:485] T f0c1c05110894e39b41e6b5c5894b8d8 P 13d05ebdca8c4f489c597ce550dd773d -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 07:59:40.709142 6511 consensus_peers.cc:462] T 0b24f67cf1a64debb23ee6618c820d33 P 3b164cbbc00f4e8eb3254ddd349a3ff9 -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111)
W20240501 07:59:40.719297 6642 consensus_peers.cc:462] T a29720541b7b4a79aaa810592ef7c4b1 P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111)
W20240501 07:59:40.751863 8859 consensus_peers.cc:485] T b4a2db0198774bb2b4f3df769260e32f P 13d05ebdca8c4f489c597ce550dd773d -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 07:59:40.754482 6642 consensus_peers.cc:485] T 6daebcfc41754ec4aca939ca20588854 P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 07:59:40.785104 8859 consensus_peers.cc:485] T c16b7962c18d4e02b850c7c7c678eae4 P 13d05ebdca8c4f489c597ce550dd773d -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 07:59:40.894400 8859 consensus_peers.cc:485] T f788de4df5454d00ac9bb883f0abc9bd P 13d05ebdca8c4f489c597ce550dd773d -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 07:59:40.950042 8728 consensus_peers.cc:485] T e5bd3a8f7a7640c09774f781009eb19c P 74077b61d6f54342bf9736b4a9f2eade -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 07:59:40.961589 8728 consensus_peers.cc:485] T 0f9e2549fe294a93b3b25081a15b2028 P 74077b61d6f54342bf9736b4a9f2eade -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 07:59:40.977880 6511 consensus_peers.cc:485] T af718dc43add4edfac703013655699f4 P 3b164cbbc00f4e8eb3254ddd349a3ff9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 07:59:41.161651 6511 consensus_peers.cc:462] T 0b24f67cf1a64debb23ee6618c820d33 P 3b164cbbc00f4e8eb3254ddd349a3ff9 -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111)
W20240501 07:59:41.193007 6249 consensus_peers.cc:485] T b7d57b7fd5234f50afbc859bc997f3e1 P 8b8b3ad3aa8f4053b2167dd550f2086f -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 07:59:41.210095 6380 consensus_peers.cc:485] T 72aa4dc7664b48b7a73d34feba7d44d3 P 8315d22c56f7439a9e1fc82f153f018c -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 07:59:41.277911 6642 consensus_peers.cc:462] T a29720541b7b4a79aaa810592ef7c4b1 P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111)
W20240501 07:59:41.329489 6380 consensus_peers.cc:485] T c3a9036ccfee45498bca9681518d3699 P 8315d22c56f7439a9e1fc82f153f018c -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 07:59:41.352206 8859 consensus_peers.cc:485] T 239ee596e62d47169eeb6670dc62d59c P 13d05ebdca8c4f489c597ce550dd773d -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 07:59:41.356458 8859 consensus_peers.cc:485] T ff9dc384dbac47dc9795617c64688943 P 13d05ebdca8c4f489c597ce550dd773d -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 07:59:41.358227 6249 consensus_peers.cc:485] T 6c3c0d6fe41f4ae196ae29beb9b68125 P 8b8b3ad3aa8f4053b2167dd550f2086f -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 07:59:41.438696 8859 consensus_peers.cc:485] T c197ba256dc74e6d92e34799c274b4e3 P 13d05ebdca8c4f489c597ce550dd773d -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 07:59:41.471403 8859 consensus_peers.cc:485] T fb6b46072a8d46e18b53b9f8de702859 P 13d05ebdca8c4f489c597ce550dd773d -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 07:59:41.539391 8859 consensus_peers.cc:485] T bab84387a40d40c3b8af7f09eb462886 P 13d05ebdca8c4f489c597ce550dd773d -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 07:59:41.577922 6511 consensus_peers.cc:485] T 114b6dbafa1c4c44ac42f1d2293e3125 P 3b164cbbc00f4e8eb3254ddd349a3ff9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 07:59:41.587929 6249 consensus_peers.cc:485] T 5ee738571af740c1b494d4f17413d1d8 P 8b8b3ad3aa8f4053b2167dd550f2086f -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 07:59:41.592941 8859 consensus_peers.cc:485] T c197ba256dc74e6d92e34799c274b4e3 P 13d05ebdca8c4f489c597ce550dd773d -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240501 07:59:41.596668 14293 raft_consensus.cc:482] T 1de93edea7714491ae339db8cdfec090 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a505e7cb133c4392bfda8a4a20874ae9)
I20240501 07:59:41.596832 14293 raft_consensus.cc:504] T 1de93edea7714491ae339db8cdfec090 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } }
I20240501 07:59:41.597326 14293 leader_election.cc:288] T 1de93edea7714491ae339db8cdfec090 P 8315d22c56f7439a9e1fc82f153f018c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149), a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607)
W20240501 07:59:41.598582 6380 leader_election.cc:334] T 1de93edea7714491ae339db8cdfec090 P 8315d22c56f7439a9e1fc82f153f018c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111)
W20240501 07:59:41.598718 6380 leader_election.cc:334] T 1de93edea7714491ae339db8cdfec090 P 8315d22c56f7439a9e1fc82f153f018c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111)
I20240501 07:59:41.598764 6380 leader_election.cc:302] T 1de93edea7714491ae339db8cdfec090 P 8315d22c56f7439a9e1fc82f153f018c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8315d22c56f7439a9e1fc82f153f018c; no voters: 83f20e84af4c40d09a4804b6b26e697f, a505e7cb133c4392bfda8a4a20874ae9
I20240501 07:59:41.598893 14296 raft_consensus.cc:2726] T 1de93edea7714491ae339db8cdfec090 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 07:59:41.623770 6642 consensus_peers.cc:485] T 3b9b71d44f754353b2321e8c36dd4dfd P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 07:59:41.647369 6380 consensus_peers.cc:485] T f88fe033bce14bf0b1b63a52a7ba137e P 8315d22c56f7439a9e1fc82f153f018c -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 07:59:41.683257 6511 consensus_peers.cc:485] T 6e2ac8c8857d45b88d3b51147d2609aa P 3b164cbbc00f4e8eb3254ddd349a3ff9 -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 07:59:41.690779 6642 consensus_peers.cc:485] T 404bb81b9c404dd288d2f8160a4273da P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 07:59:41.756630 6511 consensus_peers.cc:462] T 0b24f67cf1a64debb23ee6618c820d33 P 3b164cbbc00f4e8eb3254ddd349a3ff9 -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111)
W20240501 07:59:41.763883 8859 consensus_peers.cc:485] T 8f36eefcda3f475d9571e92fdeb52eb4 P 13d05ebdca8c4f489c597ce550dd773d -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 07:59:41.790896 6380 consensus_peers.cc:485] T 4664b04b23294ad4b0a0e5c87fdca2ad P 8315d22c56f7439a9e1fc82f153f018c -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 07:59:41.823216 6249 consensus_peers.cc:485] T 2e540f389b2a4aef93a020f549d11b67 P 8b8b3ad3aa8f4053b2167dd550f2086f -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240501 07:59:41.860226 14299 raft_consensus.cc:482] T 4d4f47308d174a92bc73f23f2242be9a P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83f20e84af4c40d09a4804b6b26e697f)
I20240501 07:59:41.860383 14299 raft_consensus.cc:504] T 4d4f47308d174a92bc73f23f2242be9a P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { replace: true } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { promote: true } }
I20240501 07:59:41.860854 14299 leader_election.cc:288] T 4d4f47308d174a92bc73f23f2242be9a P 74077b61d6f54342bf9736b4a9f2eade [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149), a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607)
W20240501 07:59:41.861999 8728 leader_election.cc:334] T 4d4f47308d174a92bc73f23f2242be9a P 74077b61d6f54342bf9736b4a9f2eade [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111)
W20240501 07:59:41.862286 8728 leader_election.cc:334] T 4d4f47308d174a92bc73f23f2242be9a P 74077b61d6f54342bf9736b4a9f2eade [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111)
I20240501 07:59:41.862365 8728 leader_election.cc:302] T 4d4f47308d174a92bc73f23f2242be9a P 74077b61d6f54342bf9736b4a9f2eade [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74077b61d6f54342bf9736b4a9f2eade; no voters: 83f20e84af4c40d09a4804b6b26e697f, a505e7cb133c4392bfda8a4a20874ae9
I20240501 07:59:41.862643 14299 raft_consensus.cc:2726] T 4d4f47308d174a92bc73f23f2242be9a P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 07:59:41.866084 6642 consensus_peers.cc:462] T a29720541b7b4a79aaa810592ef7c4b1 P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111)
W20240501 07:59:41.899698 6642 consensus_peers.cc:485] T 4bc915ffbbf345cbbe1195f8dd3ab874 P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 07:59:41.922353 6642 consensus_peers.cc:485] T 8454f24b0c28451b8ee51be39ca3e07f P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 07:59:41.990949 6249 consensus_peers.cc:485] T 0fc1ce00a2024e4a9c19abcc7d181efd P 8b8b3ad3aa8f4053b2167dd550f2086f -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 07:59:42.024979 6642 consensus_peers.cc:485] T 2560b1ec81ac4357b32c7a8ba9d72216 P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 07:59:42.082003 8859 consensus_peers.cc:485] T e0583fa43994452ca7553cbf2309eba1 P 13d05ebdca8c4f489c597ce550dd773d -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 07:59:42.092485 6380 consensus_peers.cc:485] T 9461ad0f65e1489ca102f2558194ce8f P 8315d22c56f7439a9e1fc82f153f018c -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 07:59:42.209056 8728 consensus_peers.cc:485] T ab1666f1e5204416adb435d9bbe80433 P 74077b61d6f54342bf9736b4a9f2eade -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 07:59:42.210616 6249 consensus_peers.cc:485] T f68dbc1f65eb4f1b91a62e2de82a7676 P 8b8b3ad3aa8f4053b2167dd550f2086f -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 07:59:42.222376 6642 consensus_peers.cc:485] T a29720541b7b4a79aaa810592ef7c4b1 P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 07:59:42.224598 8728 consensus_peers.cc:485] T c5a876dfb598428ab350593129505504 P 74077b61d6f54342bf9736b4a9f2eade -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 07:59:42.229452 6511 consensus_peers.cc:485] T 3745acee2b5543b697a3c8f45d2618bb P 3b164cbbc00f4e8eb3254ddd349a3ff9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 07:59:42.254360 6511 consensus_peers.cc:462] T 0b24f67cf1a64debb23ee6618c820d33 P 3b164cbbc00f4e8eb3254ddd349a3ff9 -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111)
W20240501 07:59:42.302443 6380 consensus_peers.cc:485] T 782e37a4ebb34a68b461a74115ae8703 P 8315d22c56f7439a9e1fc82f153f018c -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 07:59:42.305151 6642 consensus_peers.cc:462] T a29720541b7b4a79aaa810592ef7c4b1 P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111)
W20240501 07:59:42.307668 6380 consensus_peers.cc:485] T bf806039d8c742d8a80ad994829bdf18 P 8315d22c56f7439a9e1fc82f153f018c -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 07:59:42.319007 6511 consensus_peers.cc:485] T d3f2250a62c6429e886753aaffcbe883 P 3b164cbbc00f4e8eb3254ddd349a3ff9 -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 07:59:42.327461 6380 consensus_peers.cc:485] T a756205204d146ce82a2799e30582f57 P 8315d22c56f7439a9e1fc82f153f018c -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 07:59:42.342248 6380 consensus_peers.cc:485] T d726533fb72d4827991e12034b42634e P 8315d22c56f7439a9e1fc82f153f018c -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 07:59:42.345021 8728 consensus_peers.cc:485] T fcb3198c1fa5449781dc84756fe02bff P 74077b61d6f54342bf9736b4a9f2eade -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 07:59:42.382607 8859 consensus_peers.cc:485] T e0583fa43994452ca7553cbf2309eba1 P 13d05ebdca8c4f489c597ce550dd773d -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 07:59:42.443477 8728 consensus_peers.cc:485] T 36aaf192042f4320b2526322424dc640 P 74077b61d6f54342bf9736b4a9f2eade -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 07:59:42.473840 6249 consensus_peers.cc:485] T 1c32cd00bc344ae1b161d3f9b43b53d3 P 8b8b3ad3aa8f4053b2167dd550f2086f -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 07:59:42.477469 8728 consensus_peers.cc:485] T 6dc929e2a6ae45bd8b1f297583636bb6 P 74077b61d6f54342bf9736b4a9f2eade -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 07:59:42.583694 8728 consensus_peers.cc:485] T f3375300a7914fa599fe0d3a8f1f9c18 P 74077b61d6f54342bf9736b4a9f2eade -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 07:59:42.673285 6380 consensus_peers.cc:485] T cfcc814f4d2d488ba745ecaf0cc6d61a P 8315d22c56f7439a9e1fc82f153f018c -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 07:59:42.683291 6380 consensus_peers.cc:485] T 782e37a4ebb34a68b461a74115ae8703 P 8315d22c56f7439a9e1fc82f153f018c -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 07:59:42.708282 6511 consensus_peers.cc:485] T d3f2250a62c6429e886753aaffcbe883 P 3b164cbbc00f4e8eb3254ddd349a3ff9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 07:59:42.721432 6380 consensus_peers.cc:485] T bf806039d8c742d8a80ad994829bdf18 P 8315d22c56f7439a9e1fc82f153f018c -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 07:59:42.764463 6642 consensus_peers.cc:462] T a29720541b7b4a79aaa810592ef7c4b1 P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111)
W20240501 07:59:42.787041 8859 consensus_peers.cc:485] T ad3e528eaff349a4b3b03037652b1505 P 13d05ebdca8c4f489c597ce550dd773d -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 07:59:42.815989 6511 consensus_peers.cc:485] T d6c28cd91f624673b8e428f471a1dcce P 3b164cbbc00f4e8eb3254ddd349a3ff9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 07:59:42.826308 6511 consensus_peers.cc:462] T 0b24f67cf1a64debb23ee6618c820d33 P 3b164cbbc00f4e8eb3254ddd349a3ff9 -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111)
W20240501 07:59:42.881832 8859 consensus_peers.cc:485] T 0864c290fafa488abefa93d41c0a2efe P 13d05ebdca8c4f489c597ce550dd773d -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 07:59:42.909103 8859 consensus_peers.cc:485] T 8a39f168c8f846cd9b8c4e2f2cf62fdd P 13d05ebdca8c4f489c597ce550dd773d -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 07:59:43.069854 8859 consensus_peers.cc:485] T 12ce6a21d06c4ba889542758be9dc62d P 13d05ebdca8c4f489c597ce550dd773d -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 07:59:43.164680 6642 consensus_peers.cc:462] T a29720541b7b4a79aaa810592ef7c4b1 P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111)
W20240501 07:59:43.212713 8859 consensus_peers.cc:485] T f0c1c05110894e39b41e6b5c5894b8d8 P 13d05ebdca8c4f489c597ce550dd773d -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 07:59:43.229734 6642 consensus_peers.cc:485] T 28eaf22342ff4b5394f958d0b659273c P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 07:59:43.281172 6511 consensus_peers.cc:462] T 0b24f67cf1a64debb23ee6618c820d33 P 3b164cbbc00f4e8eb3254ddd349a3ff9 -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111)
W20240501 07:59:43.287570 8859 consensus_peers.cc:485] T b4a2db0198774bb2b4f3df769260e32f P 13d05ebdca8c4f489c597ce550dd773d -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 07:59:43.296993 8728 consensus_peers.cc:485] T e5bd3a8f7a7640c09774f781009eb19c P 74077b61d6f54342bf9736b4a9f2eade -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 07:59:43.334347 8859 consensus_peers.cc:485] T c16b7962c18d4e02b850c7c7c678eae4 P 13d05ebdca8c4f489c597ce550dd773d -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240501 07:59:43.353148 14305 raft_consensus.cc:482] T 2976d6353e8947b7b6c37f72efbeeac4 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83f20e84af4c40d09a4804b6b26e697f)
I20240501 07:59:43.353302 14305 raft_consensus.cc:504] T 2976d6353e8947b7b6c37f72efbeeac4 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "3b164cbbc00f4e8eb3254ddd349a3ff9" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 34471 } attrs { promote: false } }
I20240501 07:59:43.353762 14305 leader_election.cc:288] T 2976d6353e8947b7b6c37f72efbeeac4 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607), 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149)
W20240501 07:59:43.354921 6511 leader_election.cc:334] T 2976d6353e8947b7b6c37f72efbeeac4 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111)
W20240501 07:59:43.355019 6511 leader_election.cc:334] T 2976d6353e8947b7b6c37f72efbeeac4 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111)
I20240501 07:59:43.355047 6511 leader_election.cc:302] T 2976d6353e8947b7b6c37f72efbeeac4 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b164cbbc00f4e8eb3254ddd349a3ff9; no voters: 83f20e84af4c40d09a4804b6b26e697f, a505e7cb133c4392bfda8a4a20874ae9
I20240501 07:59:43.355171 14305 raft_consensus.cc:2726] T 2976d6353e8947b7b6c37f72efbeeac4 P 3b164cbbc00f4e8eb3254ddd349a3ff9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 07:59:43.379388 6511 consensus_peers.cc:485] T af718dc43add4edfac703013655699f4 P 3b164cbbc00f4e8eb3254ddd349a3ff9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 07:59:43.406348 8728 consensus_peers.cc:485] T 0f9e2549fe294a93b3b25081a15b2028 P 74077b61d6f54342bf9736b4a9f2eade -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 07:59:43.410692 6642 consensus_peers.cc:485] T 6daebcfc41754ec4aca939ca20588854 P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 07:59:43.464270 6380 consensus_peers.cc:485] T c3a9036ccfee45498bca9681518d3699 P 8315d22c56f7439a9e1fc82f153f018c -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 07:59:43.478989 8859 consensus_peers.cc:485] T f788de4df5454d00ac9bb883f0abc9bd P 13d05ebdca8c4f489c597ce550dd773d -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 07:59:43.568732 6642 consensus_peers.cc:462] T a29720541b7b4a79aaa810592ef7c4b1 P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111)
W20240501 07:59:43.709405 6511 consensus_peers.cc:462] T 0b24f67cf1a64debb23ee6618c820d33 P 3b164cbbc00f4e8eb3254ddd349a3ff9 -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111)
W20240501 07:59:43.776701 6642 consensus_peers.cc:485] T 3b9b71d44f754353b2321e8c36dd4dfd P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 07:59:43.799604 6380 consensus_peers.cc:485] T 72aa4dc7664b48b7a73d34feba7d44d3 P 8315d22c56f7439a9e1fc82f153f018c -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 07:59:43.835752 8859 consensus_peers.cc:485] T 239ee596e62d47169eeb6670dc62d59c P 13d05ebdca8c4f489c597ce550dd773d -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 07:59:43.840637 6249 consensus_peers.cc:485] T b7d57b7fd5234f50afbc859bc997f3e1 P 8b8b3ad3aa8f4053b2167dd550f2086f -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 07:59:43.892273 8859 consensus_peers.cc:485] T bab84387a40d40c3b8af7f09eb462886 P 13d05ebdca8c4f489c597ce550dd773d -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 07:59:43.926761 8859 consensus_peers.cc:485] T c197ba256dc74e6d92e34799c274b4e3 P 13d05ebdca8c4f489c597ce550dd773d -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 07:59:43.965469 6249 consensus_peers.cc:485] T 6c3c0d6fe41f4ae196ae29beb9b68125 P 8b8b3ad3aa8f4053b2167dd550f2086f -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 07:59:43.994553 8859 consensus_peers.cc:485] T ff9dc384dbac47dc9795617c64688943 P 13d05ebdca8c4f489c597ce550dd773d -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 07:59:44.066980 8859 consensus_peers.cc:485] T fb6b46072a8d46e18b53b9f8de702859 P 13d05ebdca8c4f489c597ce550dd773d -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 07:59:44.074774 6642 consensus_peers.cc:462] T a29720541b7b4a79aaa810592ef7c4b1 P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111)
W20240501 07:59:44.083431 6511 consensus_peers.cc:485] T 114b6dbafa1c4c44ac42f1d2293e3125 P 3b164cbbc00f4e8eb3254ddd349a3ff9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 07:59:44.123926 6380 consensus_peers.cc:485] T f88fe033bce14bf0b1b63a52a7ba137e P 8315d22c56f7439a9e1fc82f153f018c -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 07:59:44.142691 6511 consensus_peers.cc:462] T 0b24f67cf1a64debb23ee6618c820d33 P 3b164cbbc00f4e8eb3254ddd349a3ff9 -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111)
W20240501 07:59:44.175441 6642 consensus_peers.cc:485] T 404bb81b9c404dd288d2f8160a4273da P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 07:59:44.180167 8859 consensus_peers.cc:485] T 8f36eefcda3f475d9571e92fdeb52eb4 P 13d05ebdca8c4f489c597ce550dd773d -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 07:59:44.186087 6249 consensus_peers.cc:485] T 2e540f389b2a4aef93a020f549d11b67 P 8b8b3ad3aa8f4053b2167dd550f2086f -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 07:59:44.309307 6380 consensus_peers.cc:485] T 4664b04b23294ad4b0a0e5c87fdca2ad P 8315d22c56f7439a9e1fc82f153f018c -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 07:59:44.310720 6642 consensus_peers.cc:485] T a29720541b7b4a79aaa810592ef7c4b1 P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 07:59:44.327399 6511 consensus_peers.cc:485] T 6e2ac8c8857d45b88d3b51147d2609aa P 3b164cbbc00f4e8eb3254ddd349a3ff9 -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 07:59:44.346467 6249 consensus_peers.cc:485] T 5ee738571af740c1b494d4f17413d1d8 P 8b8b3ad3aa8f4053b2167dd550f2086f -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 07:59:44.348752 8859 consensus_peers.cc:485] T c197ba256dc74e6d92e34799c274b4e3 P 13d05ebdca8c4f489c597ce550dd773d -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 07:59:44.379142 6642 consensus_peers.cc:485] T 8454f24b0c28451b8ee51be39ca3e07f P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 07:59:44.443181 8859 consensus_peers.cc:485] T e0583fa43994452ca7553cbf2309eba1 P 13d05ebdca8c4f489c597ce550dd773d -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 07:59:44.460726 6642 consensus_peers.cc:462] T a29720541b7b4a79aaa810592ef7c4b1 P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111)
W20240501 07:59:44.485538 6642 consensus_peers.cc:485] T 4bc915ffbbf345cbbe1195f8dd3ab874 P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 07:59:44.564168 6380 consensus_peers.cc:485] T 9461ad0f65e1489ca102f2558194ce8f P 8315d22c56f7439a9e1fc82f153f018c -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 07:59:44.569011 6380 consensus_peers.cc:485] T bf806039d8c742d8a80ad994829bdf18 P 8315d22c56f7439a9e1fc82f153f018c -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 07:59:44.598544 6249 consensus_peers.cc:485] T 0fc1ce00a2024e4a9c19abcc7d181efd P 8b8b3ad3aa8f4053b2167dd550f2086f -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 07:59:44.617413 6511 consensus_peers.cc:462] T 0b24f67cf1a64debb23ee6618c820d33 P 3b164cbbc00f4e8eb3254ddd349a3ff9 -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111)
W20240501 07:59:44.644719 8728 consensus_peers.cc:485] T ab1666f1e5204416adb435d9bbe80433 P 74077b61d6f54342bf9736b4a9f2eade -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 07:59:44.645421 6642 consensus_peers.cc:485] T 2560b1ec81ac4357b32c7a8ba9d72216 P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 07:59:44.659651 8728 consensus_peers.cc:485] T c5a876dfb598428ab350593129505504 P 74077b61d6f54342bf9736b4a9f2eade -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240501 07:59:44.674530 14319 raft_consensus.cc:482] T 7dbbfc5d5848486f983cffdd2f36cc91 P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:44.674666 14319 raft_consensus.cc:504] T 7dbbfc5d5848486f983cffdd2f36cc91 P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } }
I20240501 07:59:44.675036 14319 leader_election.cc:288] T 7dbbfc5d5848486f983cffdd2f36cc91 P 74077b61d6f54342bf9736b4a9f2eade [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149), a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607)
W20240501 07:59:44.676144 8728 leader_election.cc:334] T 7dbbfc5d5848486f983cffdd2f36cc91 P 74077b61d6f54342bf9736b4a9f2eade [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111)
W20240501 07:59:44.676256 8728 leader_election.cc:334] T 7dbbfc5d5848486f983cffdd2f36cc91 P 74077b61d6f54342bf9736b4a9f2eade [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111)
I20240501 07:59:44.676306 8728 leader_election.cc:302] T 7dbbfc5d5848486f983cffdd2f36cc91 P 74077b61d6f54342bf9736b4a9f2eade [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74077b61d6f54342bf9736b4a9f2eade; no voters: 83f20e84af4c40d09a4804b6b26e697f, a505e7cb133c4392bfda8a4a20874ae9
I20240501 07:59:44.676429 14319 raft_consensus.cc:2726] T 7dbbfc5d5848486f983cffdd2f36cc91 P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 07:59:44.718181 6511 consensus_peers.cc:485] T d3f2250a62c6429e886753aaffcbe883 P 3b164cbbc00f4e8eb3254ddd349a3ff9 -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 07:59:44.723567 8859 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111) [suppressed 171 similar messages]
W20240501 07:59:44.731009 8859 consensus_peers.cc:485] T e0583fa43994452ca7553cbf2309eba1 P 13d05ebdca8c4f489c597ce550dd773d -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 07:59:44.738523 6511 consensus_peers.cc:485] T 3745acee2b5543b697a3c8f45d2618bb P 3b164cbbc00f4e8eb3254ddd349a3ff9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 07:59:44.778759 8728 consensus_peers.cc:485] T 36aaf192042f4320b2526322424dc640 P 74077b61d6f54342bf9736b4a9f2eade -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 07:59:44.794660 6380 consensus_peers.cc:485] T d726533fb72d4827991e12034b42634e P 8315d22c56f7439a9e1fc82f153f018c -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 07:59:44.800372 6249 consensus_peers.cc:485] T f68dbc1f65eb4f1b91a62e2de82a7676 P 8b8b3ad3aa8f4053b2167dd550f2086f -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 07:59:44.804634 6380 consensus_peers.cc:485] T a756205204d146ce82a2799e30582f57 P 8315d22c56f7439a9e1fc82f153f018c -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 07:59:44.845301 8728 consensus_peers.cc:485] T fcb3198c1fa5449781dc84756fe02bff P 74077b61d6f54342bf9736b4a9f2eade -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 07:59:44.848819 6511 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111) [suppressed 83 similar messages]
W20240501 07:59:44.856528 6380 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111) [suppressed 121 similar messages]
W20240501 07:59:44.877346 6642 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111) [suppressed 93 similar messages]
W20240501 07:59:44.878016 6642 consensus_peers.cc:462] T a29720541b7b4a79aaa810592ef7c4b1 P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111)
W20240501 07:59:44.935726 8728 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111) [suppressed 87 similar messages]
W20240501 07:59:44.936336 8728 consensus_peers.cc:485] T 6dc929e2a6ae45bd8b1f297583636bb6 P 74077b61d6f54342bf9736b4a9f2eade -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 07:59:44.943760 6380 consensus_peers.cc:485] T 782e37a4ebb34a68b461a74115ae8703 P 8315d22c56f7439a9e1fc82f153f018c -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 07:59:44.956758 6249 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240501 07:59:44.977140 6249 consensus_peers.cc:485] T 1c32cd00bc344ae1b161d3f9b43b53d3 P 8b8b3ad3aa8f4053b2167dd550f2086f -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 07:59:45.028614 6380 consensus_peers.cc:485] T 782e37a4ebb34a68b461a74115ae8703 P 8315d22c56f7439a9e1fc82f153f018c -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 07:59:45.109429 6511 consensus_peers.cc:485] T d3f2250a62c6429e886753aaffcbe883 P 3b164cbbc00f4e8eb3254ddd349a3ff9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 07:59:45.230782 6380 consensus_peers.cc:485] T bf806039d8c742d8a80ad994829bdf18 P 8315d22c56f7439a9e1fc82f153f018c -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 07:59:45.241068 6511 consensus_peers.cc:462] T 0b24f67cf1a64debb23ee6618c820d33 P 3b164cbbc00f4e8eb3254ddd349a3ff9 -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111)
W20240501 07:59:45.246686 8859 consensus_peers.cc:485] T ad3e528eaff349a4b3b03037652b1505 P 13d05ebdca8c4f489c597ce550dd773d -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 07:59:45.303578 8728 consensus_peers.cc:485] T f3375300a7914fa599fe0d3a8f1f9c18 P 74077b61d6f54342bf9736b4a9f2eade -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 07:59:45.343641 8859 consensus_peers.cc:485] T 8a39f168c8f846cd9b8c4e2f2cf62fdd P 13d05ebdca8c4f489c597ce550dd773d -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 07:59:45.372620 8859 consensus_peers.cc:485] T 0864c290fafa488abefa93d41c0a2efe P 13d05ebdca8c4f489c597ce550dd773d -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 07:59:45.385121 6380 consensus_peers.cc:485] T cfcc814f4d2d488ba745ecaf0cc6d61a P 8315d22c56f7439a9e1fc82f153f018c -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 07:59:45.465684 6642 consensus_peers.cc:462] T a29720541b7b4a79aaa810592ef7c4b1 P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111)
W20240501 07:59:45.508867 8859 consensus_peers.cc:485] T 12ce6a21d06c4ba889542758be9dc62d P 13d05ebdca8c4f489c597ce550dd773d -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 07:59:45.561313 8859 consensus_peers.cc:485] T b4a2db0198774bb2b4f3df769260e32f P 13d05ebdca8c4f489c597ce550dd773d -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 07:59:45.621424 6511 consensus_peers.cc:485] T d6c28cd91f624673b8e428f471a1dcce P 3b164cbbc00f4e8eb3254ddd349a3ff9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 07:59:45.635691 8859 consensus_peers.cc:485] T f0c1c05110894e39b41e6b5c5894b8d8 P 13d05ebdca8c4f489c597ce550dd773d -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 07:59:45.661823 8728 consensus_peers.cc:485] T e5bd3a8f7a7640c09774f781009eb19c P 74077b61d6f54342bf9736b4a9f2eade -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 07:59:45.679040 6642 consensus_peers.cc:485] T 28eaf22342ff4b5394f958d0b659273c P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 07:59:45.707628 6642 consensus_peers.cc:485] T 6daebcfc41754ec4aca939ca20588854 P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 07:59:45.814666 6511 consensus_peers.cc:462] T 0b24f67cf1a64debb23ee6618c820d33 P 3b164cbbc00f4e8eb3254ddd349a3ff9 -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111)
W20240501 07:59:45.887905 8859 consensus_peers.cc:485] T c16b7962c18d4e02b850c7c7c678eae4 P 13d05ebdca8c4f489c597ce550dd773d -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 07:59:45.891875 8728 consensus_peers.cc:485] T 0f9e2549fe294a93b3b25081a15b2028 P 74077b61d6f54342bf9736b4a9f2eade -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 07:59:45.925660 8859 consensus_peers.cc:485] T f788de4df5454d00ac9bb883f0abc9bd P 13d05ebdca8c4f489c597ce550dd773d -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 07:59:45.965817 6642 consensus_peers.cc:462] T a29720541b7b4a79aaa810592ef7c4b1 P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111)
W20240501 07:59:46.000392 6511 consensus_peers.cc:485] T af718dc43add4edfac703013655699f4 P 3b164cbbc00f4e8eb3254ddd349a3ff9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 07:59:46.053766 6249 consensus_peers.cc:485] T b7d57b7fd5234f50afbc859bc997f3e1 P 8b8b3ad3aa8f4053b2167dd550f2086f -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 07:59:46.095189 6380 consensus_peers.cc:485] T c3a9036ccfee45498bca9681518d3699 P 8315d22c56f7439a9e1fc82f153f018c -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 07:59:46.153831 6380 consensus_peers.cc:485] T 72aa4dc7664b48b7a73d34feba7d44d3 P 8315d22c56f7439a9e1fc82f153f018c -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 07:59:46.160333 8859 consensus_peers.cc:485] T 239ee596e62d47169eeb6670dc62d59c P 13d05ebdca8c4f489c597ce550dd773d -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 07:59:46.200647 6511 consensus_peers.cc:462] T 0b24f67cf1a64debb23ee6618c820d33 P 3b164cbbc00f4e8eb3254ddd349a3ff9 -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111)
W20240501 07:59:46.258313 8859 consensus_peers.cc:485] T bab84387a40d40c3b8af7f09eb462886 P 13d05ebdca8c4f489c597ce550dd773d -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 07:59:46.357738 8859 consensus_peers.cc:485] T c197ba256dc74e6d92e34799c274b4e3 P 13d05ebdca8c4f489c597ce550dd773d -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 07:59:46.433588 6642 consensus_peers.cc:462] T a29720541b7b4a79aaa810592ef7c4b1 P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111)
W20240501 07:59:46.460032 6249 consensus_peers.cc:485] T 2e540f389b2a4aef93a020f549d11b67 P 8b8b3ad3aa8f4053b2167dd550f2086f -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 07:59:46.505896 6380 consensus_peers.cc:485] T f88fe033bce14bf0b1b63a52a7ba137e P 8315d22c56f7439a9e1fc82f153f018c -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 07:59:46.517449 6249 consensus_peers.cc:485] T 6c3c0d6fe41f4ae196ae29beb9b68125 P 8b8b3ad3aa8f4053b2167dd550f2086f -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 07:59:46.537890 6642 consensus_peers.cc:485] T 404bb81b9c404dd288d2f8160a4273da P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 07:59:46.541648 8859 consensus_peers.cc:485] T fb6b46072a8d46e18b53b9f8de702859 P 13d05ebdca8c4f489c597ce550dd773d -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 07:59:46.555335 8859 consensus_peers.cc:485] T e0583fa43994452ca7553cbf2309eba1 P 13d05ebdca8c4f489c597ce550dd773d -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 07:59:46.580981 8859 consensus_peers.cc:485] T ff9dc384dbac47dc9795617c64688943 P 13d05ebdca8c4f489c597ce550dd773d -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 07:59:46.616171 6511 consensus_peers.cc:485] T 114b6dbafa1c4c44ac42f1d2293e3125 P 3b164cbbc00f4e8eb3254ddd349a3ff9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 07:59:46.621464 8859 consensus_peers.cc:485] T 8f36eefcda3f475d9571e92fdeb52eb4 P 13d05ebdca8c4f489c597ce550dd773d -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 07:59:46.622618 6642 consensus_peers.cc:485] T 3b9b71d44f754353b2321e8c36dd4dfd P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 07:59:46.715562 6511 consensus_peers.cc:485] T 6e2ac8c8857d45b88d3b51147d2609aa P 3b164cbbc00f4e8eb3254ddd349a3ff9 -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 07:59:46.805295 6511 consensus_peers.cc:462] T 0b24f67cf1a64debb23ee6618c820d33 P 3b164cbbc00f4e8eb3254ddd349a3ff9 -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111)
W20240501 07:59:46.900979 6642 consensus_peers.cc:485] T a29720541b7b4a79aaa810592ef7c4b1 P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 07:59:46.923936 8859 consensus_peers.cc:485] T c197ba256dc74e6d92e34799c274b4e3 P 13d05ebdca8c4f489c597ce550dd773d -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 07:59:46.928232 6380 consensus_peers.cc:485] T 4664b04b23294ad4b0a0e5c87fdca2ad P 8315d22c56f7439a9e1fc82f153f018c -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 07:59:46.931788 6249 consensus_peers.cc:485] T 5ee738571af740c1b494d4f17413d1d8 P 8b8b3ad3aa8f4053b2167dd550f2086f -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 07:59:46.983000 6380 consensus_peers.cc:485] T bf806039d8c742d8a80ad994829bdf18 P 8315d22c56f7439a9e1fc82f153f018c -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 07:59:47.008563 6642 consensus_peers.cc:485] T 8454f24b0c28451b8ee51be39ca3e07f P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 07:59:47.021833 6642 consensus_peers.cc:462] T a29720541b7b4a79aaa810592ef7c4b1 P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111)
W20240501 07:59:47.022423 6642 consensus_peers.cc:485] T 4bc915ffbbf345cbbe1195f8dd3ab874 P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 07:59:47.095767 6249 consensus_peers.cc:485] T 0fc1ce00a2024e4a9c19abcc7d181efd P 8b8b3ad3aa8f4053b2167dd550f2086f -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 07:59:47.116753 6249 consensus_peers.cc:485] T f68dbc1f65eb4f1b91a62e2de82a7676 P 8b8b3ad3aa8f4053b2167dd550f2086f -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 07:59:47.123000 6511 consensus_peers.cc:485] T 3745acee2b5543b697a3c8f45d2618bb P 3b164cbbc00f4e8eb3254ddd349a3ff9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 07:59:47.158834 6642 consensus_peers.cc:485] T 2560b1ec81ac4357b32c7a8ba9d72216 P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 07:59:47.184696 6380 consensus_peers.cc:485] T 9461ad0f65e1489ca102f2558194ce8f P 8315d22c56f7439a9e1fc82f153f018c -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 07:59:47.190518 8728 consensus_peers.cc:485] T 36aaf192042f4320b2526322424dc640 P 74077b61d6f54342bf9736b4a9f2eade -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 07:59:47.193228 6511 consensus_peers.cc:485] T d3f2250a62c6429e886753aaffcbe883 P 3b164cbbc00f4e8eb3254ddd349a3ff9 -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 07:59:47.203980 8728 consensus_peers.cc:485] T c5a876dfb598428ab350593129505504 P 74077b61d6f54342bf9736b4a9f2eade -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 07:59:47.251821 6380 consensus_peers.cc:485] T a756205204d146ce82a2799e30582f57 P 8315d22c56f7439a9e1fc82f153f018c -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 07:59:47.265954 6511 consensus_peers.cc:462] T 0b24f67cf1a64debb23ee6618c820d33 P 3b164cbbc00f4e8eb3254ddd349a3ff9 -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111)
W20240501 07:59:47.273734 8728 consensus_peers.cc:485] T ab1666f1e5204416adb435d9bbe80433 P 74077b61d6f54342bf9736b4a9f2eade -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 07:59:47.332895 8728 consensus_peers.cc:485] T fcb3198c1fa5449781dc84756fe02bff P 74077b61d6f54342bf9736b4a9f2eade -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 07:59:47.344709 6380 consensus_peers.cc:485] T 782e37a4ebb34a68b461a74115ae8703 P 8315d22c56f7439a9e1fc82f153f018c -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 07:59:47.388468 6380 consensus_peers.cc:485] T d726533fb72d4827991e12034b42634e P 8315d22c56f7439a9e1fc82f153f018c -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 07:59:47.404163 8859 consensus_peers.cc:485] T e0583fa43994452ca7553cbf2309eba1 P 13d05ebdca8c4f489c597ce550dd773d -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 07:59:47.444501 8728 consensus_peers.cc:485] T 6dc929e2a6ae45bd8b1f297583636bb6 P 74077b61d6f54342bf9736b4a9f2eade -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240501 07:59:47.460124 14293 raft_consensus.cc:482] T 13daf8496cd64d7d959ce5ad666b9be8 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a505e7cb133c4392bfda8a4a20874ae9)
I20240501 07:59:47.460309 14293 raft_consensus.cc:504] T 13daf8496cd64d7d959ce5ad666b9be8 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } }
I20240501 07:59:47.460686 14293 leader_election.cc:288] T 13daf8496cd64d7d959ce5ad666b9be8 P 8315d22c56f7439a9e1fc82f153f018c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149), a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607)
W20240501 07:59:47.461900 6380 leader_election.cc:334] T 13daf8496cd64d7d959ce5ad666b9be8 P 8315d22c56f7439a9e1fc82f153f018c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111)
W20240501 07:59:47.462040 6380 leader_election.cc:334] T 13daf8496cd64d7d959ce5ad666b9be8 P 8315d22c56f7439a9e1fc82f153f018c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111)
I20240501 07:59:47.462093 6380 leader_election.cc:302] T 13daf8496cd64d7d959ce5ad666b9be8 P 8315d22c56f7439a9e1fc82f153f018c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8315d22c56f7439a9e1fc82f153f018c; no voters: 83f20e84af4c40d09a4804b6b26e697f, a505e7cb133c4392bfda8a4a20874ae9
I20240501 07:59:47.462272 14293 raft_consensus.cc:2726] T 13daf8496cd64d7d959ce5ad666b9be8 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 07:59:47.472397 6642 consensus_peers.cc:462] T a29720541b7b4a79aaa810592ef7c4b1 P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111)
W20240501 07:59:47.500344 6249 consensus_peers.cc:485] T 1c32cd00bc344ae1b161d3f9b43b53d3 P 8b8b3ad3aa8f4053b2167dd550f2086f -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 07:59:47.522753 6380 consensus_peers.cc:485] T bf806039d8c742d8a80ad994829bdf18 P 8315d22c56f7439a9e1fc82f153f018c -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 07:59:47.588534 8859 consensus_peers.cc:485] T 0864c290fafa488abefa93d41c0a2efe P 13d05ebdca8c4f489c597ce550dd773d -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 07:59:47.591164 6380 consensus_peers.cc:485] T 782e37a4ebb34a68b461a74115ae8703 P 8315d22c56f7439a9e1fc82f153f018c -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 07:59:47.647607 6511 consensus_peers.cc:462] T 0b24f67cf1a64debb23ee6618c820d33 P 3b164cbbc00f4e8eb3254ddd349a3ff9 -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111)
W20240501 07:59:47.665271 8859 consensus_peers.cc:485] T ad3e528eaff349a4b3b03037652b1505 P 13d05ebdca8c4f489c597ce550dd773d -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 07:59:47.671449 6511 consensus_peers.cc:485] T d3f2250a62c6429e886753aaffcbe883 P 3b164cbbc00f4e8eb3254ddd349a3ff9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 07:59:47.842200 8728 consensus_peers.cc:485] T f3375300a7914fa599fe0d3a8f1f9c18 P 74077b61d6f54342bf9736b4a9f2eade -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 07:59:47.857144 8859 consensus_peers.cc:485] T 8a39f168c8f846cd9b8c4e2f2cf62fdd P 13d05ebdca8c4f489c597ce550dd773d -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 07:59:47.949218 8859 consensus_peers.cc:485] T b4a2db0198774bb2b4f3df769260e32f P 13d05ebdca8c4f489c597ce550dd773d -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 07:59:48.024139 6642 consensus_peers.cc:485] T 28eaf22342ff4b5394f958d0b659273c P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 07:59:48.040516 6511 consensus_peers.cc:485] T d6c28cd91f624673b8e428f471a1dcce P 3b164cbbc00f4e8eb3254ddd349a3ff9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 07:59:48.079236 8859 consensus_peers.cc:485] T 12ce6a21d06c4ba889542758be9dc62d P 13d05ebdca8c4f489c597ce550dd773d -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 07:59:48.084455 6642 consensus_peers.cc:462] T a29720541b7b4a79aaa810592ef7c4b1 P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111)
W20240501 07:59:48.096832 6380 consensus_peers.cc:485] T cfcc814f4d2d488ba745ecaf0cc6d61a P 8315d22c56f7439a9e1fc82f153f018c -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 07:59:48.104144 6642 consensus_peers.cc:485] T 6daebcfc41754ec4aca939ca20588854 P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 07:59:48.151093 8728 consensus_peers.cc:485] T 0f9e2549fe294a93b3b25081a15b2028 P 74077b61d6f54342bf9736b4a9f2eade -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 07:59:48.151193 8728 consensus_peers.cc:485] T e5bd3a8f7a7640c09774f781009eb19c P 74077b61d6f54342bf9736b4a9f2eade -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 07:59:48.220414 6511 consensus_peers.cc:485] T af718dc43add4edfac703013655699f4 P 3b164cbbc00f4e8eb3254ddd349a3ff9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 07:59:48.246280 6511 consensus_peers.cc:462] T 0b24f67cf1a64debb23ee6618c820d33 P 3b164cbbc00f4e8eb3254ddd349a3ff9 -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111)
W20240501 07:59:48.252566 8859 consensus_peers.cc:485] T f0c1c05110894e39b41e6b5c5894b8d8 P 13d05ebdca8c4f489c597ce550dd773d -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 07:59:48.291687 8859 consensus_peers.cc:485] T f788de4df5454d00ac9bb883f0abc9bd P 13d05ebdca8c4f489c597ce550dd773d -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 07:59:48.295468 8859 consensus_peers.cc:485] T c16b7962c18d4e02b850c7c7c678eae4 P 13d05ebdca8c4f489c597ce550dd773d -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240501 07:59:48.437330 14335 raft_consensus.cc:482] T 7dbbfc5d5848486f983cffdd2f36cc91 P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 07:59:48.437464 14335 raft_consensus.cc:504] T 7dbbfc5d5848486f983cffdd2f36cc91 P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } }
I20240501 07:59:48.437912 14335 leader_election.cc:288] T 7dbbfc5d5848486f983cffdd2f36cc91 P 74077b61d6f54342bf9736b4a9f2eade [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149), a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607)
W20240501 07:59:48.439114 8728 leader_election.cc:334] T 7dbbfc5d5848486f983cffdd2f36cc91 P 74077b61d6f54342bf9736b4a9f2eade [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111)
W20240501 07:59:48.439136 6380 consensus_peers.cc:485] T c3a9036ccfee45498bca9681518d3699 P 8315d22c56f7439a9e1fc82f153f018c -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 07:59:48.439257 8728 leader_election.cc:334] T 7dbbfc5d5848486f983cffdd2f36cc91 P 74077b61d6f54342bf9736b4a9f2eade [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111)
I20240501 07:59:48.439312 8728 leader_election.cc:302] T 7dbbfc5d5848486f983cffdd2f36cc91 P 74077b61d6f54342bf9736b4a9f2eade [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74077b61d6f54342bf9736b4a9f2eade; no voters: 83f20e84af4c40d09a4804b6b26e697f, a505e7cb133c4392bfda8a4a20874ae9
I20240501 07:59:48.439446 14335 raft_consensus.cc:2726] T 7dbbfc5d5848486f983cffdd2f36cc91 P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 07:59:48.469939 6642 consensus_peers.cc:462] T a29720541b7b4a79aaa810592ef7c4b1 P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111)
W20240501 07:59:48.682637 6511 consensus_peers.cc:462] T 0b24f67cf1a64debb23ee6618c820d33 P 3b164cbbc00f4e8eb3254ddd349a3ff9 -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111)
W20240501 07:59:48.692129 6380 consensus_peers.cc:485] T 72aa4dc7664b48b7a73d34feba7d44d3 P 8315d22c56f7439a9e1fc82f153f018c -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 07:59:48.708364 8859 consensus_peers.cc:485] T 239ee596e62d47169eeb6670dc62d59c P 13d05ebdca8c4f489c597ce550dd773d -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240501 07:59:48.716230 14327 raft_consensus.cc:482] T c86c420310ec42ffa5b24304fba74267 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83f20e84af4c40d09a4804b6b26e697f)
I20240501 07:59:48.716378 14327 raft_consensus.cc:504] T c86c420310ec42ffa5b24304fba74267 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { replace: true } } peers { permanent_uuid: "8315d22c56f7439a9e1fc82f153f018c" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 32941 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39859 } attrs { promote: true } }
I20240501 07:59:48.716774 14327 leader_election.cc:288] T c86c420310ec42ffa5b24304fba74267 P 8315d22c56f7439a9e1fc82f153f018c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149), a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607)
W20240501 07:59:48.718328 6380 leader_election.cc:334] T c86c420310ec42ffa5b24304fba74267 P 8315d22c56f7439a9e1fc82f153f018c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111)
W20240501 07:59:48.718541 6380 leader_election.cc:334] T c86c420310ec42ffa5b24304fba74267 P 8315d22c56f7439a9e1fc82f153f018c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111)
I20240501 07:59:48.718605 6380 leader_election.cc:302] T c86c420310ec42ffa5b24304fba74267 P 8315d22c56f7439a9e1fc82f153f018c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8315d22c56f7439a9e1fc82f153f018c; no voters: 83f20e84af4c40d09a4804b6b26e697f, a505e7cb133c4392bfda8a4a20874ae9
I20240501 07:59:48.718737 14327 raft_consensus.cc:2726] T c86c420310ec42ffa5b24304fba74267 P 8315d22c56f7439a9e1fc82f153f018c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 07:59:48.749971 8859 consensus_peers.cc:485] T c197ba256dc74e6d92e34799c274b4e3 P 13d05ebdca8c4f489c597ce550dd773d -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 07:59:48.760509 6249 consensus_peers.cc:485] T 6c3c0d6fe41f4ae196ae29beb9b68125 P 8b8b3ad3aa8f4053b2167dd550f2086f -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 07:59:48.776878 8859 consensus_peers.cc:485] T bab84387a40d40c3b8af7f09eb462886 P 13d05ebdca8c4f489c597ce550dd773d -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 07:59:48.788331 6249 consensus_peers.cc:485] T b7d57b7fd5234f50afbc859bc997f3e1 P 8b8b3ad3aa8f4053b2167dd550f2086f -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 07:59:48.886498 6380 consensus_peers.cc:485] T f88fe033bce14bf0b1b63a52a7ba137e P 8315d22c56f7439a9e1fc82f153f018c -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 07:59:48.887784 6249 consensus_peers.cc:485] T 2e540f389b2a4aef93a020f549d11b67 P 8b8b3ad3aa8f4053b2167dd550f2086f -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 07:59:48.899646 8859 consensus_peers.cc:485] T fb6b46072a8d46e18b53b9f8de702859 P 13d05ebdca8c4f489c597ce550dd773d -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 07:59:48.918682 8859 consensus_peers.cc:485] T 8f36eefcda3f475d9571e92fdeb52eb4 P 13d05ebdca8c4f489c597ce550dd773d -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 07:59:48.925382 8859 consensus_peers.cc:485] T ff9dc384dbac47dc9795617c64688943 P 13d05ebdca8c4f489c597ce550dd773d -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 07:59:49.032290 6642 consensus_peers.cc:462] T a29720541b7b4a79aaa810592ef7c4b1 P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111)
W20240501 07:59:49.074542 6511 consensus_peers.cc:485] T 114b6dbafa1c4c44ac42f1d2293e3125 P 3b164cbbc00f4e8eb3254ddd349a3ff9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 07:59:49.084110 6380 consensus_peers.cc:485] T 4664b04b23294ad4b0a0e5c87fdca2ad P 8315d22c56f7439a9e1fc82f153f018c -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 07:59:49.157192 6642 consensus_peers.cc:485] T 404bb81b9c404dd288d2f8160a4273da P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 07:59:49.193979 6158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111) [suppressed 8 similar messages]
W20240501 07:59:49.194782 6158 catalog_manager.cc:4573] TS 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): CreateTablet RPC failed for tablet 7dbbfc5d5848486f983cffdd2f36cc91: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111)
W20240501 07:59:49.194876 6158 catalog_manager.cc:4605] Request timed out: CreateTablet RPC for tablet 7dbbfc5d5848486f983cffdd2f36cc91 on TS 83f20e84af4c40d09a4804b6b26e697f
I20240501 07:59:49.252103 14335 raft_consensus.cc:482] T a93ce5841c7f40a9ab23f88efe7b85bb P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83f20e84af4c40d09a4804b6b26e697f)
I20240501 07:59:49.252249 14335 raft_consensus.cc:504] T a93ce5841c7f40a9ab23f88efe7b85bb P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "83f20e84af4c40d09a4804b6b26e697f" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 46149 } } peers { permanent_uuid: "74077b61d6f54342bf9736b4a9f2eade" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39859 } } peers { permanent_uuid: "a505e7cb133c4392bfda8a4a20874ae9" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 42607 } attrs { replace: true } } peers { permanent_uuid: "8b8b3ad3aa8f4053b2167dd550f2086f" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 46875 } attrs { promote: true } }
W20240501 07:59:49.252467 6511 consensus_peers.cc:462] T 0b24f67cf1a64debb23ee6618c820d33 P 3b164cbbc00f4e8eb3254ddd349a3ff9 -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111)
I20240501 07:59:49.252677 14335 leader_election.cc:288] T a93ce5841c7f40a9ab23f88efe7b85bb P 74077b61d6f54342bf9736b4a9f2eade [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149), a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607)
W20240501 07:59:49.253621 8728 leader_election.cc:334] T a93ce5841c7f40a9ab23f88efe7b85bb P 74077b61d6f54342bf9736b4a9f2eade [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111)
W20240501 07:59:49.253721 8728 leader_election.cc:334] T a93ce5841c7f40a9ab23f88efe7b85bb P 74077b61d6f54342bf9736b4a9f2eade [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111)
I20240501 07:59:49.253769 8728 leader_election.cc:302] T a93ce5841c7f40a9ab23f88efe7b85bb P 74077b61d6f54342bf9736b4a9f2eade [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74077b61d6f54342bf9736b4a9f2eade; no voters: 83f20e84af4c40d09a4804b6b26e697f, a505e7cb133c4392bfda8a4a20874ae9
I20240501 07:59:49.253973 14335 raft_consensus.cc:2726] T a93ce5841c7f40a9ab23f88efe7b85bb P 74077b61d6f54342bf9736b4a9f2eade [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 07:59:49.357218 6511 consensus_peers.cc:485] T 6e2ac8c8857d45b88d3b51147d2609aa P 3b164cbbc00f4e8eb3254ddd349a3ff9 -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 07:59:49.362159 6642 consensus_peers.cc:485] T 4bc915ffbbf345cbbe1195f8dd3ab874 P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 07:59:49.398396 8859 consensus_peers.cc:485] T e0583fa43994452ca7553cbf2309eba1 P 13d05ebdca8c4f489c597ce550dd773d -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 07:59:49.421130 6642 consensus_peers.cc:485] T 3b9b71d44f754353b2321e8c36dd4dfd P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 07:59:49.432415 8859 consensus_peers.cc:485] T c197ba256dc74e6d92e34799c274b4e3 P 13d05ebdca8c4f489c597ce550dd773d -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 07:59:49.448423 6380 consensus_peers.cc:485] T bf806039d8c742d8a80ad994829bdf18 P 8315d22c56f7439a9e1fc82f153f018c -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 07:59:49.471123 6642 consensus_peers.cc:485] T 8454f24b0c28451b8ee51be39ca3e07f P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 07:59:49.492038 6642 consensus_peers.cc:462] T a29720541b7b4a79aaa810592ef7c4b1 P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111)
F20240501 07:59:49.495338 9240 rebalancer_tool-test.cc:1121] Check failed: _s.ok() Bad status: Timed out: Timed out waiting for AlterTable
*** Check failure stack trace: ***
*** Aborted at 1714550389 (unix time) try "date -d @1714550389" if you are using GNU date ***
PC: @ 0x0 (unknown)
*** SIGABRT (@0x3e80000018e) received by PID 398 (TID 0x7fadf7fbd700) from PID 398; stack trace: ***
@ 0x7fae0dfa9980 (unknown) at ??:0
@ 0x7fae0b57ffb7 gsignal at ??:0
@ 0x7fae0b581921 abort at ??:0
@ 0x7fae0c7bfdcd google::LogMessage::Fail() at ??:0
@ 0x7fae0c7c3b93 google::LogMessage::SendToLog() at ??:0
@ 0x7fae0c7bf7cc google::LogMessage::Flush() at ??:0
@ 0x7fae0c7c0f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x557d828fc057 _ZZN4kudu5tools70DDLDuringRebalancingTest_TablesCreatedAndDeletedDuringRebalancing_Test8TestBodyEvENKUlvE3_clEv at ??:0
@ 0x557d8291556e _ZSt13__invoke_implIvZN4kudu5tools70DDLDuringRebalancingTest_TablesCreatedAndDeletedDuringRebalancing_Test8TestBodyEvEUlvE3_JEET_St14__invoke_otherOT0_DpOT1_ at ??:0
@ 0x557d829142db _ZSt8__invokeIZN4kudu5tools70DDLDuringRebalancingTest_TablesCreatedAndDeletedDuringRebalancing_Test8TestBodyEvEUlvE3_JEENSt15__invoke_resultIT_JDpT0_EE4typeEOS5_DpOS6_ at ??:0
@ 0x557d8291bcdc _ZNSt6thread8_InvokerISt5tupleIJZN4kudu5tools70DDLDuringRebalancingTest_TablesCreatedAndDeletedDuringRebalancing_Test8TestBodyEvEUlvE3_EEE9_M_invokeIJLm0EEEEDTcl8__invokespcl10_S_declvalIXT_EEEEESt12_Index_tupleIJXspT_EEE at ??:0
@ 0x557d8291bb65 _ZNSt6thread8_InvokerISt5tupleIJZN4kudu5tools70DDLDuringRebalancingTest_TablesCreatedAndDeletedDuringRebalancing_Test8TestBodyEvEUlvE3_EEEclEv at ??:0
@ 0x557d8291ba30 _ZNSt6thread11_State_implINS_8_InvokerISt5tupleIJZN4kudu5tools70DDLDuringRebalancingTest_TablesCreatedAndDeletedDuringRebalancing_Test8TestBodyEvEUlvE3_EEEEE6_M_runEv at ??:0
@ 0x7fae0bfa56df (unknown) at ??:0
@ 0x7fae0df9e6db start_thread at ??:0
@ 0x7fae0b66271f clone at ??:0
W20240501 07:59:49.532944 6249 consensus_peers.cc:485] T 5ee738571af740c1b494d4f17413d1d8 P 8b8b3ad3aa8f4053b2167dd550f2086f -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 07:59:49.560482 8728 consensus_peers.cc:485] T c5a876dfb598428ab350593129505504 P 74077b61d6f54342bf9736b4a9f2eade -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 07:59:49.560511 6642 consensus_peers.cc:485] T a29720541b7b4a79aaa810592ef7c4b1 P 1f9cb72be98941b5a8e504cbe34132b9 -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 07:59:49.663797 6249 consensus_peers.cc:485] T f68dbc1f65eb4f1b91a62e2de82a7676 P 8b8b3ad3aa8f4053b2167dd550f2086f -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 07:59:49.673875 6511 consensus_peers.cc:485] T d3f2250a62c6429e886753aaffcbe883 P 3b164cbbc00f4e8eb3254ddd349a3ff9 -> Peer 83f20e84af4c40d09a4804b6b26e697f (127.0.99.136:46149): Couldn't send request to peer 83f20e84af4c40d09a4804b6b26e697f. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.136:46149: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 07:59:49.686183 8728 consensus_peers.cc:485] T fcb3198c1fa5449781dc84756fe02bff P 74077b61d6f54342bf9736b4a9f2eade -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 07:59:49.714247 6380 consensus_peers.cc:485] T a756205204d146ce82a2799e30582f57 P 8315d22c56f7439a9e1fc82f153f018c -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 07:59:49.725080 6380 consensus_peers.cc:485] T 9461ad0f65e1489ca102f2558194ce8f P 8315d22c56f7439a9e1fc82f153f018c -> Peer a505e7cb133c4392bfda8a4a20874ae9 (127.0.99.135:42607): Couldn't send request to peer a505e7cb133c4392bfda8a4a20874ae9. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:42607: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.